Moving image encoding device, moving image decoding device, moving image coding method, and moving image decoding method

ABSTRACT

When an encoding mode corresponding to one of blocks to be encoded into which an image is divided by a block dividing part  2  is an inter encoding mode which is a direct mode, a motion-compensated prediction part  5  selects a motion vector suitable for generation of a prediction image from one or more selectable motion vectors and also carries out a motion-compensated prediction process on the block to be encoded to generate a prediction image by using the motion vector, and outputs index information showing the motion vector to a variable length encoding part  13,  and the variable length encoding unit  13  variable-length-encodes the index information.

This application is a Divisional of copending application Ser. No.13/824,279, filed on Mar. 15, 2013, which claims priority under 35U.S.C. §119(a) to Application No. PCT/JP2011/004121, filed on Jul. 21,2011, which claims priority under 35 U.S.C. §119(a) to Application No.JP2011-050214, filed in Japan on Mar. 8, 2011, and Application No.JP2010-221460, filed in Japan on Sep. 30, 2010, all of which are herebyexpressly incorporated by reference into the present application.

FIELD OF THE INVENTION

The present invention relates to a moving image encoding device, amoving image decoding device, a moving image encoding method, and amoving image decoding method which are used for an image compressionencoding technology, a compressed image data transmission technology,etc.

BACKGROUND OF THE INVENTION

For example, in an international standard video encoding system, such asMPEG (Moving Picture Experts Group) or “ITU-T H.26x”, a method ofdefining block data (referred to as a “macroblock” from here on) whichis a combination of 16×16 pixels for a luminance signal and 8×8 pixelsfor each of color difference signals which correspond to the 16×16pixels of the luminance signal as one unit, and compressing image dataon the basis of a motion compensation technology and an orthogonaltransformation/transform coefficient quantization technology is used. Inmotion compensation processes carried out by a moving image encodingdevice and a moving image decoding device, a forward picture or abackward picture is referred to, and detection of a motion vector andgeneration of a prediction image are carried out for each macroblock. Atthis time, a picture for which only one picture is referred to and onwhich inter-frame prediction encoding is carried out is referred to as aP picture, and a picture for which two pictures is simultaneouslyreferred to and on which inter-frame prediction encoding is carried outis referred to as a B picture.

In AVC/H.264 which is an international standard system (ISO/IEC14496-101 ITU-T H.264), an encoding mode called a direct mode can beselected when encoding a B picture (for example, refer to nonpatentreference 1). More specifically, a macroblock to be encoded does nothave encoded data of a motion vector, and an encoding mode in which togenerate a motion vector of the macroblock to be encoded can be selectedin a predetermined arithmetic process using a motion vector of amacroblock of another already-encoded picture and a motion vector of anadjacent macroblock.

This direct mode includes the following two types of modes: a temporaldirect mode and a spatial direct mode. In the temporal direct mode, byreferring to the motion vector of another already-encoded picture andthen carrying out a scaling process of scaling the motion vectoraccording to the time difference between the other already-encodedpicture and the picture which is the target to be encoded, a motionvector of the macroblock to be encoded is generated. In the spatialdirect mode, by referring to the motion vector of at least onealready-encoded macroblock located in the vicinity of the macroblock tobe encoded, a motion vector of the macroblock to be encoded is generatedfrom the motion vector. In this direct mode, either of the temporaldirect mode and the spatial direct mode can be selected for each sliceby using “direct_spatial_mv_pred_flag” which is a flag disposed in eachslice header. A mode in which transform coefficients are not encoded,among direct modes, is referred to as a skip mode. Hereafter, a skipmode is also included in a direct mode which will be described below.

FIG. 11 is a schematic diagram showing a method of generating a motionvector in the temporal direct mode. In FIG. 11, “P” denotes a P pictureand “B” denotes a B picture. Further, numerical numbers 0 to 3 denote anorder in which pictures respectively designated by the numerical numbersare displayed, and show images which are displayed at times T0, T1, T2,and T3, respectively. It is assumed that an encoding process on thepictures is carried out in order of P0, P3, B1, and B2.

For example, a case in which a macroblock MB1 in the picture B2 isencoded in the temporal direct mode will be considered hereafter. Inthis case, the motion vector MV of a macroblock MB2 which is a motionvector of the picture P3 closest to the picture B2 among thealready-encoded pictures located backward with respect to the picture B2on the time axis, and which is spatially located at the same position asthe macroblock MB1. This motion vector MV refers to the picture P0, andmotion vectors MVL0 and MVL1 which are used when encoding the macroblockMB1 are calculated according to the following equation (1).

$\begin{matrix}{{{{MVL}\; 0} = {\frac{{T\; 2} - {T\; 0}}{{T\; 3} - {T\; 0}} \times M\; V}}{{{MVL}\; 1} = {\frac{{T\; 2} - {T\; 3}}{{T\; 3} - {T\; 0}} \times M\; V}}} & (1)\end{matrix}$

FIG. 12 is a schematic diagram showing a method of generating a motionvector in the spatial direct mode. In FIG. 12, currentMB denotes themacroblock to be encoded. At this time, when the motion vector of analready-encoded macroblock A on a left side of the macroblock to beencoded is expressed as MVa, the motion vector of an already-encodedmacroblock B on an upper side of the macroblock to be encoded isexpressed as MVb, and the motion vector of an already-encoded macroblockC on an upper right side of the macroblock to be encoded is expressed asMVc, the motion vector MV of the macroblock to be encoded can becalculated by determining the median of these motion vectors MVa, MVb,and MVc, as shown in the following equation (2).

MV=median(MVa, MVb, MVc)   (2)

The motion vector is determined for each of forward and backwardpictures in the spatial direct mode, and the motion vectors for both ofthem can be determined by using the above-mentioned method.

A reference image which is used for the generation of a prediction imageis managed as a reference image list for each vector which is used forreference. When two vectors are used, reference image lists are referredto as a list 0 and a list 1, respectively. Reference images are storedin the reference image lists in reverse chronological order,respectively, and, in a general case, the list 0 shows a forwardreference image and the list 1 shows a backward reference image. As analternative, the list 1 can show a forward reference image and the list0 can show a backward reference image, or each of the lists 0 and 1 canshow a forward reference image and a backward reference image. Further,the reference image lists do not have to be aligned in reversechronological order.

For example, the following nonpatent reference 1 describes that thereference image lists can be ordered for each slice.

RELATED ART DOCUMENT Nonpatent Reference

Nonpatent reference 1: MPEG-4 AVC (ISO/IEC 14496-10)/H.ITU-T 264standards

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Because the conventional image encoding device is constructed as above,the conventional image encoding device can switch between the temporaldirect mode and the spatial direct mode on a per slice basis by simplyreferring to “direct_spatial_mv_pred_flag” which is a flag disposed ineach slice header. However, because the conventional image encodingdevice cannot switch between the temporal direct mode and the spatialdirect mode on a per macroblock basis, even though an optimal directmode for a macroblock belonging to a slice is the spatial direct mode,for example, the conventional image encoding device has to use thetemporal direct mode for the macroblock when the direct modecorresponding to the slice is determined to be the temporal direct mode,and therefore cannot select the optimal direct mode. In such a case,because the conventional image encoding device cannot select the optimaldirect mode, the conventional image encoding device has to encode anunnecessary motion vector and there arises a problem of increase in thecode amount.

The present invention is made in order to solve the above-mentionedproblem, and it is therefore an object of the present invention toprovide a moving image encoding device, a moving image decoding device,a moving image encoding method, and a moving image decoding methodcapable of selecting an optimal direct mode for each predetermined blockunit, thereby being able to reduce the code amount.

Means for Solving the Problem

In accordance with the present invention, there is provided a movingimage encoding device including: an encoding controlling unit fordetermining a maximum size of a block to be encoded which is a unit tobe processed when a prediction process is carried out, and alsodetermining a maximum hierarchy depth when a block to be encoded havingthe maximum size is divided hierarchically, and for selecting anencoding mode which determines an encoding method of encoding each blockto be encoded from one or more available encoding modes; and a blockdividing unit for dividing an inputted image into blocks to be encodedhaving a predetermined size, and also dividing each of theabove-mentioned blocks to be encoded hierarchically, in which when aninter encoding mode which is a direct mode is selected by the encodingcontrolling unit as an encoding mode corresponding to one of the blocksto be encoded into which the inputted image is divided by the blockdividing unit, a motion-compensated prediction unit selects a motionvector suitable for generation of a prediction image from one or moreselectable motion vectors and also carries out a motion-compensatedprediction process on the above-mentioned block to be encoded togenerate a prediction image by using the motion vector, and outputsindex information showing the motion vector to a variable lengthencoding unit, and the variable length encoding unitvariable-length-encoding the index information.

Advantages of the Invention

Because the moving image encoding device in accordance with the presentinvention is constructed in such a way as that the moving image encodingdevice includes: the encoding controlling unit for determining a maximumsize of a block to be encoded which is a unit to be processed when aprediction process is carried out, and also determining a maximumhierarchy depth when a block to be encoded having the maximum size isdivided hierarchically, and for selecting an encoding mode whichdetermines an encoding method of encoding each block to be encoded fromone or more available encoding modes; and the block dividing unit fordividing an inputted image into blocks to be encoded having apredetermined size, and also dividing each of the above-mentioned blocksto be encoded hierarchically and, when an inter encoding mode which is adirect mode is selected by the encoding controlling unit as an encodingmode corresponding to one of the blocks to be encoded into which theinputted image is divided by the block dividing unit, themotion-compensated prediction unit selects a motion vector suitable forgeneration of a prediction image from one or more selectable motionvectors and also carries out a motion-compensated prediction process onthe above-mentioned block to be encoded to generate a prediction imageby using the motion vector, and outputs index information showing themotion vector to the variable length encoding unit, and the variablelength encoding unit variable-length-encoding the index information,there is provided an advantage of being able to select an optimal directmode for each predetermined block unit, and reduce the code amount.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram showing a moving image encoding device inaccordance with Embodiment 1 of the present invention;

FIG. 2 is a block diagram showing a motion-compensated prediction part 5of the moving image encoding device in accordance with Embodiment 1 ofthe present invention;

FIG. 3 is a block diagram showing a direct vector generating part 23which constructs the motion-compensated prediction part 5;

FIG. 4 is a block diagram showing a direct vector determining part 33which constructs the direct vector generating part 23;

FIG. 5 is a block diagram showing a moving image decoding device inaccordance with Embodiment 1 of the present invention;

FIG. 6 is a block diagram showing a motion-compensated prediction part54 of the moving image decoding device in accordance with Embodiment 1of the present invention;

FIG. 7 is a flow chart showing processing carried out by the movingimage encoding device in accordance with Embodiment 1 of the presentinvention;

FIG. 8 is a flow chart showing processing carried out by the movingimage decoding device in accordance with Embodiment 1 of the presentinvention;

FIG. 9 is an explanatory drawing showing a state in which each block tobe encoded having a maximum size is hierarchically divided into aplurality of blocks to be encoded;

FIG. 10( a) is an explanatory drawing showing a distribution ofpartitions into which a block to encoded is divided, and FIG. 10( b) isan explanatory drawing showing a state in which an encoding modem(B^(n)) is assigned to each of the partitions after a hierarchicallayer division is performed by using a quadtree graph;

FIG. 11 is a schematic diagram showing a method of generating a motionvector in a temporal direct mode;

FIG. 12 is a schematic diagram showing a method of generating a motionvector in a spatial direct mode;

FIG. 13 is a schematic diagram showing a method of generating a spatialdirect vector from candidates A1 to An, B1 to B^(n), C, D, and E formedian prediction;

FIG. 14 is a schematic diagram showing a method of generating a spatialdirect vector by carrying out scaling according to a distance in atemporal direction;

FIG. 15 is an explanatory drawing showing an example of calculation ofan evaluated value based on the degree of similarity between a forwardprediction image and a backward prediction image;

FIGS. 16 a & 16 b are explanatory drawings showing an evaluationequation using a variance of motion vectors;

FIG. 17 is an explanatory drawing showing spatial vectors MV_A, MV_B,and MV_C, and temporal vectors MV_(—)1 to MV_(—)8;

FIG. 18 is an explanatory drawing showing generation of one candidatevector from a plurality of already-encoded vectors;

FIG. 19 is an explanatory drawing showing an example of calculating anevaluated value SAD from a combination of only images located backwardin time;

FIG. 20 is an explanatory drawing showing a search for an image similarto an L-shaped template;

FIG. 21 is an explanatory drawing showing an example in which the sizeof a block to be encoded B^(n) is L^(n)=kM^(n);

FIG. 22 is an explanatory drawing showing an example of a divisionsatisfying (L^(n+1), M^(n+1))=(L^(n)/2, M^(n)/2);

FIG. 23 is an explanatory drawing showing an example in which a divisionshown in either FIG. 21 or FIG. 22 can be selected;

FIG. 24 is an explanatory drawing showing an example in which atransformation block size unit has a hierarchical structure;

FIG. 25 is a block diagram showing a motion-compensated prediction part5 of a moving image encoding device in accordance with Embodiment 3 ofthe present invention;

FIG. 26 is a block diagram showing a direct vector generation part 25which constructs the motion-compensated prediction part 5;

FIG. 27 is a block diagram showing an initial vector generating part 34which constructs the direct vector generation part 25;

FIG. 28 is a block diagram showing an initial vector determining part 73which constructs the initial vector generating part 34.

FIG. 29 is a block diagram showing a motion-compensated prediction part54 of a moving image decoding device in accordance with Embodiment 3 ofthe present invention;

FIG. 30 is an explanatory drawing showing a process of searching for amotion vector;

FIG. 31 is a block diagram showing a motion-compensated prediction part5 of a moving image encoding device in accordance with Embodiment 4 ofthe present invention;

FIG. 32 is a block diagram showing a motion-compensated prediction part54 of a moving image decoding device in accordance with Embodiment 4 ofthe present invention;

FIG. 33 is an explanatory drawing showing a direct vector candidateindex in which a selectable motion vector and index information showingthe motion vector are described;

FIG. 34 is an explanatory drawing showing an example of encoding onlyindex information showing one vector;

FIG. 35 is a block diagram showing a direct vector generation part 26which constructs the motion-compensated prediction part 5;

FIG. 36 is a block diagram showing a motion-compensated prediction part5 of a moving image encoding device in accordance with Embodiment 5 ofthe present invention;

FIG. 37 is a block diagram showing a direct vector generation part 27which constructs the motion-compensated prediction part 5;

FIG. 38 is a block diagram showing a motion-compensated prediction part54 of a moving image decoding device in accordance with Embodiment 5 ofthe present invention;

FIG. 39 is a block diagram showing a direct vector generation part 26which constructs the motion-compensated prediction part 5;

FIG. 40 is an explanatory drawing showing a correlation with an adjacentblock;

FIG. 41 is an explanatory drawing of a list showing one or moreselectable motion vectors for each of block sizes provided for blocks tobe encoded;

FIG. 42 is an explanatory drawing showing an example of a list whosemaximum block size is “128”;

FIG. 43 is an explanatory drawing of a list showing one or moreselectable motion vectors for each of division patterns provided forblocks to be encoded;

FIG. 44 is a flow chart showing a process of transmitting listinformation in a moving image encoding device;

FIG. 45 is a flow chart showing a process of receiving list informationin a moving image decoding device;

FIG. 46 is an explanatory drawing showing an example of encoding achange flag set to “ON” and list information showing a changed listbecause “temporal” in a list is changed from selectable to unselectable;

FIG. 47 is an explanatory drawing showing an example of changing a listcurrently being held because a change flag is set to “ON”;

FIG. 48 is an explanatory drawing showing an example of preparing achange flag for each block size, and encoding only list informationassociated with a block size for which selectable motion vectors arechanged; and

FIG. 49 is an explanatory drawing showing an example of searching for ablock which is inter-encoded from a target block, and setting allvectors included in the block as spatial vector candidates.

EMBODIMENTS OF THE INVENTION

Hereafter, the preferred embodiments of the present invention will beexplained in detail with reference to the drawings.

Embodiment 1

In this Embodiment 1, a moving image encoding device that inputs eachframe image of a video, carries out variable length encoding on theframe image after carrying out a compression process with an orthogonaltransformation and quantization on a prediction difference signal whichthe moving image encoding device acquires by carrying out amotion-compensated prediction between adjacent frames to generate abitstream, and a moving image decoding device that decodes the bitstreamoutputted from the moving image encoding device will be explained.

The moving image encoding device in accordance with this Embodiment 1 ischaracterized in that the moving image encoding device adapts itself toa local change of a video signal in spatial and temporal directions todivide the video signal into regions of various sizes, and carries outintra-frame and inter-frame adaptive encoding. In general, a videosignal has a characteristic of its complexity varying locally in spaceand time. There can be a case in which a pattern having a uniform signalcharacteristic in a relatively large image area, such as a sky image ora wall image, or a pattern having a complicated texture pattern in asmall image area, such as a person image or a picture including a finetexture, also coexists on a certain video frame from the viewpoint ofspace. Also from the viewpoint of time, a relatively large image area,such as a sky image or a wall image, has a small local change in atemporal direction in its pattern, while an image of a moving person orobject has a larger temporal change because its outline has a movementof a rigid body and a movement of a non-rigid body with respect to time.

Although in the encoding process a process of generating a predictiondifference signal having small signal power and small entropy by usingtemporal and spatial prediction, thereby reducing the whole code amount,is carried out, the code amount of a parameter used for the predictioncan be reduced as long as the parameter can be applied uniformly to aslarge an image signal region as possible. On the other hand, because theamount of errors occurring in the prediction increases when the sameprediction parameter is applied to an image signal pattern having alarge change in time and space, the code amount of the predictiondifference signal cannot be reduced. Therefore, it is desirable toreduce the size of a region which is subjected to the prediction processwhen performing the prediction process on an image signal pattern havinga large change in time and space, thereby reducing the electric powerand entropy of the prediction difference signal even though the datavolume of the parameter which is used for the prediction is increased.In order to carry out encoding which is adapted for such the typicalcharacteristics of a video signal, the moving image encoding device inaccordance with this Embodiment 1 hierarchically divides each regionhaving a predetermined maximum block size of the video signal intoblocks, and carries out the prediction process and the encoding processof encoding the prediction difference on each of the blocks into whicheach region is divided.

A video signal which is to be processed by the moving image encodingdevice in accordance with this Embodiment 1 can be an arbitrary videosignal in which each video frame consists of a series of digital samples(pixels) in two dimensions, horizontal and vertical, such as a YUVsignal which consists of a luminance signal and two color differencesignals, a color video image signal in arbitrary color space, such as anRGB signal, outputted from a digital image sensor, a monochrome imagesignal, or an infrared image signal. The gradation of each pixel can bean 8-bit, 10-bit, or 12-bit one. In the following explanation, theinputted video signal is a YUV signal unless otherwise specified. It isfurther assumed that the two color difference components U and V aresignals having a 4:2:0 format which are subsampled with respect to theluminance component Y. A data unit to be processed which corresponds toeach frame of the video signal is referred to as a “picture.” In thisEmbodiment 1, a “picture” is explained as a video frame signal on whichprogressive scanning is carried out. When the video signal is aninterlaced signal, a “picture” can be alternatively a field image signalwhich is a unit which constructs a video frame.

FIG. 1 is a block diagram showing the moving image encoding device inaccordance with Embodiment 1 of the present invention. Referring to FIG.1, an encoding controlling part 1 carries out a process of determining amaximum size of each of blocks to be encoded which is a unit to beprocessed at a time when a motion-compensated prediction process(inter-frame prediction process) or an intra prediction process(intra-frame prediction process) is carried out, and also determining anupper limit on the number of hierarchical layers, i.e., a maximumhierarchy depth in a hierarchy in which each of the blocks to be encodedhaving the maximum size is hierarchically divided into blocks. Theencoding controlling part 1 also carries out a process of selecting anencoding mode suitable for each of the blocks to be encoded into whicheach block to be encoded having a maximum size is divided hierarchicallyfrom among one or more available encoding modes (one or more intraencoding modes and one or more inter encoding modes (including an interencoding mode which is a direct mode)). The encoding controlling part 1constructs an encoding controlling unit.

A block dividing part 2 carries out a process of, when receiving a videosignal showing an inputted image, dividing the inputted image shown bythe video signal into blocks to be encoded each having the maximum sizedetermined by the encoding controlling part 1, and also dividing each ofthe blocks to be encoded into blocks hierarchically until the number ofhierarchical layers reaches the upper limit on the number ofhierarchical layers which is determined by the encoding controlling part1. The block dividing part 2 constructs a block dividing unit.

A selection switch 3 carries out a process of, when the encoding modeselected by the encoding controlling part 1 for the block to be encoded,which is generated through the division by the block dividing part 2, isan intra encoding mode, outputting the block to be encoded to an intraprediction part 4, and, when the encoding mode selected by the encodingcontrolling part 1 for the block to be encoded, which is generatedthrough the division by the block dividing part 2, is an inter encodingmode, outputting the block to be encoded to a motion-compensatedprediction part 5. The intra prediction part 4 carries out a process of,when receiving the block to be encoded, which is generated through thedivision by the block dividing part 2, from the selection switch 3,performing an intra prediction process on the block to be encoded byusing intra prediction parameters outputted from the encodingcontrolling part 1 to generate a prediction image. An intra predictionunit is comprised of the selection switch 3 and the intra predictionpart 4.

The motion-compensated prediction part 5 carries out a process of, whenan inter encoding mode which is a direct mode is selected by theencoding controlling part 1 as the encoding mode suitable for the blockto be encoded, which is generated through the division by the blockdividing part 2, generating a spatial direct vector in a spatial directmode from the motion vector of an already-encoded block located in thevicinity of the block to be encoded and also generating a temporaldirect vector in a temporal direct mode from the motion vector of analready-encoded picture which can be referred to by the block to beencoded, selecting a direct vector which provides a higher correlationbetween reference images from the spatial direct vector and the temporaldirect vector, and performing a motion-compensated prediction process onthe block to be encoded by using the direct vector selected thereby togenerate a prediction image. In contrast, when an inter encoding modeother than a direct mode is selected by the encoding controlling part 1as the encoding mode suitable for the block to be encoded, which isgenerated through the division by the block dividing part 2, themotion-compensated prediction part 5 carries out a process of searchingthrough the block to be encoded and a reference image stored in amotion-compensated prediction frame memory 12 for a motion vector, andperforming a motion-compensated prediction process on the block to beencoded by using the motion vector to generate a prediction image. Amotion-compensated prediction unit is comprised of the selection switch3 and the motion-compensated prediction part 5.

A subtracting part 6 carries out a process of subtracting the predictionimage generated by the intra prediction part 4 or the motion-compensatedprediction part 5 from the block to be encoded, which is generatedthrough the division by the block dividing part 2, to generate adifference image (=the block to be encoded−the prediction image). Thesubtracting part 6 constructs a difference image generating unit. Atransformation/quantization part 7 carries out a process of performingan orthogonal transformation process (e.g., a DCT (discrete cosinetransform) or an orthogonal transformation process, such as a KLtransform, in which bases are designed for a specific learning sequencein advance) on the difference signal generated by the subtracting part 6in units of a block having a transformation block size included inprediction difference encoding parameters outputted from the encodingcontrolling part 1, and also quantizing the transform coefficients ofthe difference image by using a quantization parameter included in theprediction difference encoding parameters to output the transformcoefficients quantized thereby as compressed data of the differenceimage. The transformation/quantization part 7 constructs an imagecompression unit.

An inverse quantization/inverse transformation part 8 carries out aprocess of inverse-quantizing the compressed data outputted from thetransformation/quantization part 7 by using the quantization parameterincluded in the prediction difference encoding parameter outputted fromthe encoding controlling part 1, and performing an inversetransformation process (e.g., an inverse DCT (inverse discrete cosinetransform) or an inverse transformation process such as an inverse KLtransform) on the compressed data inverse-quantized thereby to outputthe compressed data on which the inverse quantization/inversetransformation part carries out the inverse transformation process as alocal decoded prediction difference signal.

An adding part 9 carries out a process of adding the local decodedprediction difference signal outputted from the inversequantization/inverse transformation part 8 and the prediction signalshowing the prediction image generated by the intra prediction part 4 orthe motion-compensated prediction part 5 to generate a local decodedimage signal showing a local decoded image. A memory 10 for intraprediction is a recording medium, such as a RAM, for storing the localdecoded image shown by the local decoded image signal generated by theadding part 9 as an image which the intra prediction part 4 will usewhen performing the intra prediction process the next time.

A loop filter part 11 carries out a process of compensating for anencoding distortion included in the local decoded image signal generatedby the adding part 9, and outputting the local decoded image shown bythe local decoded image signal on which the loop filter part performsthe encoding distortion compensation to a motion-compensated predictionframe memory 12 as a reference image. The motion-compensated predictionframe memory 12 is a recording medium, such as a RAM, for storing thelocal decoded image on which the loop filter part 11 performs thefiltering process as a reference image which the motion-compensatedprediction part 5 will use when performing the motion-compensatedprediction process the next time.

A variable length encoding part 13 carries out a process ofvariable-length-encoding the compressed data outputted from thetransformation/quantization part 7, the encoding mode and the predictiondifference encoding parameters which are outputted from the encodingcontrolling part 1, and the intra prediction parameters outputted fromthe intra prediction part 4 or inter prediction parameters outputtedfrom the motion-compensated prediction part 5 to generate a bitstreaminto which encoded data of the compressed data, encoded data of theencoding mode, encoded data of the prediction difference encodingparameters, and encoded data of the intra prediction parameters or theinter prediction parameters are multiplexed. The variable lengthencoding part 13 constructs a variable length encoding unit.

FIG. 2 is a block diagram showing the motion-compensated prediction part5 of the moving image encoding device in accordance with Embodiment 1 ofthe present invention. Referring to FIG. 2, a selection switch 21carries out a process of outputting the block to be encoded, which isgenerated through the division by the block dividing part 2, to a motionvector searching part 22 when the encoding mode selected by the encodingcontrolling part 1 is an inter mode other than direct modes, andoutputting the block to be encoded, which is generated through thedivision by the block dividing part 2, to a direct vector generatingpart 23 when the encoding mode is an inter mode which is a direct mode.Because the direct vector generating part 23 does not use the block tobe encoded, which is generated through the division by the blockdividing part 2, when generating a direct vector, the selection switchdoes not have to output the block to be encoded to the direct vectorgenerating part 23.

The motion vector searching part 22 carries out a process of searchingfor an optimal motion vector in the inter mode while referring to boththe block to be encoded outputted from the selection switch 21 and areference image stored in the motion-compensated prediction frame memory12, and outputting the motion vector to a motion compensation processingpart 24. The direct vector generating part 23 carries out a process ofgenerating a spatial direct vector in the spatial direct mode from themotion vector of an already-encoded block located in the vicinity of theblock to be encoded, and also generating a temporal direct vector in thetemporal direct mode from the motion vector of an already-encodedpicture which can be referred to by the block to be encoded, andselecting a direct vector which provides a higher correlation betweenreference images from the spatial direct vector and the temporal directvector.

The motion compensation processing part 24 carries out a process ofperforming a motion-compensated prediction process on the basis of theinter prediction parameters outputted from the encoding controlling part1 by using both the motion vector which is searched for by the motionvector searching part 22 or the direct vector which is selected by thedirect vector generating part 23, and one or more frames of referenceimages stored in the motion-compensated prediction frame memory 12 togenerate a prediction image. The motion compensation processing part 24outputs the inter prediction parameters when the motion compensationprocessing part uses when carrying out the motion-compensated predictionprocess to the variable length encoding part 13. When the encoding modeselected by the encoding controlling part 1 is an inter mode other thandirect modes, the motion compensation processing part includes themotion vector which is searched for by the motion vector searching part22 in the inter prediction parameters, and outputs these interprediction parameters to the variable length encoding part 13.

FIG. 3 is a block diagram showing the direct vector generating part 23which constructs the motion-compensated prediction part 5. Referring toFIG. 3, a spatial direct vector generating part 31 carries out a processof reading the motion vector of an already-encoded block located in thevicinity of the block to be encoded from among the motion vectors ofalready-encoded blocks (the motion vectors of already-encoded blocks arestored in a not-shown motion vector memory or an internal memory of themotion-compensated prediction part 5) to generate a spatial directvector in a spatial direct mode from the motion vector. A temporaldirect vector generating part 32 carries out a process of reading themotion vector of a block located spatially at the same position as theblock to be encoded, which is the motion vector of an already-encodedpicture which can be referred to by the block to be encoded, from amongthe motion vectors of already-encoded blocks to generate a temporaldirect vector in the temporal direct mode from the motion vector.

A direct vector determining part 33 carries out a process of calculatingan evaluated value in the spatial direct mode by using the spatialdirect vector generated by the spatial direct vector generating part 31and also calculating an evaluated value in the temporal direct mode byusing the temporal direct vector generated by the temporal direct vectorgenerating part 32, and comparing the evaluated value in the spatialdirect mode with the evaluated value in the temporal direct mode toselect either of the spatial direct vector and the temporal directvector.

FIG. 4 is a block diagram showing the direct vector determining part 33which constructs the direct vector generating part 23. Referring to FIG.4, a motion compensation part 41 carries out a process of generating alist 0 prediction image in the spatial direct mode (e.g., a forwardprediction image in the spatial direct mode) and a list 1 predictionimage in the spatial direct mode (e.g., a backward prediction image inthe spatial direct mode) by using the spatial direct vector generated bythe spatial direct vector generating part 31, and also generating a list0 prediction image in the temporal direct mode (e.g., a forwardprediction image in the temporal direct mode) and a list 1 predictionimage in the temporal direct mode (e.g., a backward prediction image inthe temporal direct mode) by using the temporal direct vector generatedby the temporal direct vector generating part 32.

A similarity calculating part 42 carries out a process of calculatingthe degree of similarity between the list 0 prediction image in thespatial direct mode (forward prediction image) and the list 1 predictionimage in the spatial direct mode (backward prediction image) as theevaluated value in the spatial direct mode, and also calculating thedegree of similarity between the list 0 prediction image in the temporaldirect mode (forward prediction image) and the list 1 prediction imagein the temporal direct mode (backward prediction image) as the evaluatedvalue in the temporal direct mode. A direct vector selecting part 43carries out a process of comparing the degree of similarity between thelist 0 prediction image in the spatial direct mode (forward predictionimage) and the list 1 prediction image in the spatial direct mode(backward prediction image), which is calculated by the similaritycalculating part 42, with the degree of similarity between the list 0prediction image in the temporal direct mode (forward prediction image)and the list 1 prediction image in the temporal direct mode (backwardprediction image), which is calculated by the similarity calculatingpart 42, to select the direct vector in one direct mode which provides ahigher degree of similarity between the list 0 prediction image (forwardprediction image) and the list 1 prediction image (backward predictionimage) from the spatial direct vector and the temporal direct vector.

FIG. 5 is a block diagram showing a moving image decoding device inaccordance with Embodiment 1 of the present invention. Referring to FIG.5, a variable length decoding part 51 carries out a process ofvariable-length-decoding the encoded data multiplexed into the bitstreamto acquire the compressed data, the encoding mode, the predictiondifference encoding parameters, and the intra prediction parameters orthe inter prediction parameters, which are associated with each codingblock into which each frame of the video is hierarchically divided, andoutputting the compressed data and the prediction difference encodingparameters to an inverse quantization/inverse transformation part 55,and also outputting the encoding mode, and the intra predictionparameters or the inter prediction parameters to a selection switch 52.The variable length decoding part 51 constructs a variable lengthdecoding unit.

The selection switch 52 carries out a process of, when the encoding modeassociated with the coding block, which is outputted from the variablelength decoding part 51, is an intra encoding mode, outputting the intraprediction parameters outputted thereto from the variable lengthdecoding part 51 to an intra prediction part 53, and, when the encodingmode is an inter encoding mode, outputting the inter predictionparameters outputted thereto from the variable length decoding part 51to a motion-compensated prediction part 54. The intra prediction part 53carries out a process of performing an intra prediction process on thecoding block by using the intra prediction parameters outputted theretofrom the selection switch 52 to generate a prediction image. An intraprediction unit is comprised of the selection switch 52 and the intraprediction part 53.

The motion-compensated prediction part 54 carries out a process of, whenthe encoding mode associated with the coding block, which is outputtedthereto from the variable length decoding part 51, is an inter encodingmode which is a direct mode, generating a spatial direct vector in thespatial direct mode from the motion vector of an already-decoded blocklocated in the vicinity of the coding block and also generating atemporal direct vector in the temporal direct mode from the motionvector of an already-decoded picture which can be referred to by thecoding block, selecting one direct vector which provides a highercorrelation between reference images from the spatial direct vector andthe temporal direct vector, and performing a motion-compensatedprediction process on the coding block by using the direct vectorselected thereby to generate a prediction image. The motion-compensatedprediction part 54 also carries out a process of performing amotion-compensated prediction process on the coding block by using themotion vector included in the inter prediction parameters outputtedthereto from the variable length decoding part 51 to generate aprediction image when the encoding mode associated with the codingblock, which is outputted thereto from the variable length decoding part51, is an inter encoding mode other than direct modes. Amotion-compensated prediction unit is comprised of the selection switch52 and the motion-compensated prediction part 54.

An inverse quantization/inverse transformation part 55 carries out aprocess of inverse-quantizing the compressed data associated with thecoding block, which is outputted thereto from the variable lengthdecoding part 51, by using the quantization parameter included in theprediction difference encoding parameters outputted thereto from thevariable length decoding part 51, and performing an inversetransformation process (e.g., an inverse DCT (inverse discrete cosinetransform) or an inverse transformation process such as an inverse KLtransform) on the compressed data inverse-quantized thereby in units ofa block having the transformation block size included in the predictiondifference encoding parameters, and outputting the compressed data onwhich the inverse quantization/inverse transformation part performs theinverse transformation process as a decoded prediction difference signal(signal showing a pre-compressed difference image). The inversequantization/inverse transformation part 55 constructs a differenceimage generating unit.

A n adding part 56 carries out a process of adding the decodedprediction difference signal outputted thereto from the inversequantization/inverse transformation part 55 and the prediction signalshowing the prediction image generated by the intra prediction part 53or the motion-compensated prediction part 54 to generate a decoded imagesignal showing a decoded image. The adding part 56 constructs a decodedimage generating unit. A memory 57 for intra prediction is a recordingmedium, such as a RAM, for storing the decoded image shown by thedecoded image signal generated by the adding part 56 as an image whichthe intra prediction part 53 will use when performing the intraprediction process the next time.

A loop filter part 58 carries out a process of compensating for anencoding distortion included in the decoded image signal generated bythe adding part 56, and outputting the decoded image shown by thedecoded image signal on which the loop filter part performs the encodingdistortion compensation to a motion-compensated prediction frame memory59 as a reference image. The motion-compensated prediction frame memory59 is a recording medium, such as a RAM, for storing the decoded imageon which the loop filter part 58 performs the filtering process as areference image which the motion-compensated prediction part 54 will usewhen performing the motion-compensated prediction process the next time.

FIG. 6 is a block diagram showing the motion-compensated prediction part54 of the moving image decoding device in accordance with Embodiment 1of the present invention. Referring to FIG. 6, a selection switch 61carries out a process of, when the encoding mode associated with thecoding block, which is outputted thereto from the variable lengthdecoding part 51, is an inter mode other than direct modes, outputtingthe inter prediction parameters (including the motion vector) outputtedthereto from the variable length decoding part 51 to a motioncompensation processing part 63, and, when the encoding mode is an intermode which is an direct mode, outputting the inter prediction parametersoutputted thereto from the variable length decoding part 51 to a directvector generating part 62.

The direct vector generating part 62 carries out a process of generatinga spatial direct vector in the spatial direct mode from the motionvector of an already-decoded block located in the vicinity of the codingblock and also generates a temporal direct vector in the temporal directmode from the motion vector of an already-decoded picture which can bereferred to by the coding block, and selecting one direct vector whichprovides a higher correlation between reference images from the spatialdirect vector and the temporal direct vector. The direct vectorgenerating part 62 also carries out a process of outputting the interprediction parameters outputted thereto from the selection switch 61 tothe motion compensation processing part 63. The internal structure ofthe direct vector generating part 62 is the same as the direct vectorgenerating part 23 shown in FIG. 2.

The motion compensation processing part 63 carries out a process ofperforming a motion-compensated prediction process on the basis of theinter prediction parameters outputted thereto from the direct vectorgenerating part 62 by using both the motion vector included in the interprediction parameters outputted thereto from the selection switch 61 orthe direct vector selected by the direct vector generating part 62, anda reference image of one frame stored in the motion-compensatedprediction frame memory 59 to generate a prediction image.

In the example of FIG. 1, the encoding controlling part 1, the blockdividing part 2, the selection switch 3, the intra prediction part 4,the motion-compensated prediction part 5, the subtracting part 6, thetransformation/quantization part 7, the inverse quantization/inversetransformation part 8, the adding part 9, the loop filter part 11, andthe variable length encoding part 13, which are the components of themoving image encoding device, can consist of pieces of hardware forexclusive use (e.g., integrated circuits in each of which a CPU ismounted, one chip microcomputers, or the like), respectively. As analternative, the moving image encoding device can consist of a computer,and a program in which the processes carried out by the encodingcontrolling part 1, the block dividing part 2, the selection switch 3,the intra prediction part 4, the motion-compensated prediction part 5,the subtracting part 6, the transformation/quantization part 7, theinverse quantization/inverse transformation part 8, the adding part 9,the loop filter part 11, and the variable length encoding part 13 aredescribed can be stored in a memory of the computer and the CPU of thecomputer can be made to execute the program stored in the memory. FIG. 7is a flow chart showing the processing carried out by the moving imageencoding device in accordance with Embodiment 1 of the presentinvention.

In the example of FIG. 5, the variable length decoding part 51, theselection switch 52, the intra prediction part 53, themotion-compensated prediction part 54, the inverse quantization/inversetransformation part 55, the adding part 56, and the loop filter part 58,which are the components of the moving image decoding device, canconsist of pieces of hardware for exclusive use (e.g., integratedcircuits in each of which a CPU is mounted, one chip microcomputers, orthe like), respectively. As an alternative, the moving image decodingdevice can consist of a computer, and a program in which the processescarried out by the variable length decoding part 51, the selectionswitch 52, the intra prediction part 53, the motion-compensatedprediction part 54, the inverse quantization/inverse transformation part55, the adding part 56, and the loop filter part 58 are described can bestored in a memory of the computer and the CPU of the computer can bemade to execute the program stored in the memory. FIG. 8 is a flow chartshowing the processing carried out by the moving image decoding devicein accordance with Embodiment 1 of the present invention.

Next, the operation of the moving image encoding device and theoperation of the moving image decoding device will be explained. First,the processing carried out by the moving image encoding device shown inFIG. 1 will be explained. First, the encoding controlling part 1determines a maximum size of each of blocks to be encoded which is aunit to be processed at a time when a motion-compensated predictionprocess (inter-frame prediction process) or an intra prediction process(intra-frame prediction process) is carried out, and also determines anupper limit on the number of hierarchical layers in a hierarchy in whicheach of the blocks to be encoded having the maximum size ishierarchically divided into blocks (step ST1 of FIG. 7).

As a method of determining the maximum size of each of blocks to beencoded, for example, there is considered a method of determining amaximum size for all the pictures according to the resolution of theinputted image. Further, there can be considered a method of quantifyinga variation in the complexity of a local movement of the inputted imageas a parameter and then determining a small size for a picture having alarge and vigorous movement while determining a large size for a picturehaving a small movement. As a method of determining the upper limit onthe number of hierarchical layers, for example, there can be considereda method of increasing the depth of the hierarchy, i.e., the number ofhierarchical layers to make it possible to detect a finer movement asthe inputted image has a larger and more vigorous movement, ordecreasing the depth of the hierarchy, i.e., the number of hierarchicallayers as the inputted image has a smaller movement.

The encoding controlling part 1 also selects an encoding mode suitablefor each of the blocks to be encoded into which each block to be encodedhaving the maximum size is divided hierarchically from among one or moreavailable encoding modes (M intra encoding modes and N inter encodingmodes (including an inter encoding mode which is a direct mode)) (stepST2). Although a detailed explanation of the selection method ofselecting an encoding mode for use in the encoding controlling part 1will be omitted because the selection method is a known technique, thereis a method of carrying out an encoding process on the block to beencoded by using an arbitrary available encoding mode to examine theencoding efficiency and select an encoding mode having the highest levelof encoding efficiency from among a plurality of available encodingmodes, for example.

When receiving the video signal showing the inputted image, the blockdividing part 2 divides the inputted image shown by the video signalinto blocks to be encoded each having the maximum size determined by theencoding controlling part 1, and also divides each of the blocks to beencoded into blocks hierarchically until the number of hierarchicallayers reaches the upper limit on the number of hierarchical layerswhich is determined by the encoding controlling part 1. FIG. 9 is anexplanatory drawing showing a state in which each block to be encodedhaving the maximum size is hierarchically divided into a plurality ofblocks to be encoded. In the example of FIG. 9, each block to be encodedhaving the maximum size is a block to be encoded B⁰ in the 0thhierarchical layer, and its luminance component has a size of (L⁰, M⁰).Further, in the example of FIG. 9, by carrying out the hierarchicaldivision with this block to be encoded B⁰ having the maximum size beingset as a starting point until the depth of the hierarchy reaches apredetermined depth which is set separately according to a quadtreestructure, blocks to be encoded B^(n) can be acquired.

At the depth of n, each block to be encoded B^(n) is an image areahaving a size of (L^(n), M^(n)). In this example, although M^(n) can bethe same as or differ from L^(n), the case of L^(n)=M^(n) is shown inFIG. 4. Hereafter, the size of each block to be encoded B^(n) is definedas the size of (L^(n), M^(n)) in the luminance component of the block tobe encoded B^(n).

Because the block dividing part 2 carries out a quadtree division,(L^(n+1), m^(n+1))=(L^(n)/2, M^(n)/2) is always established. In the caseof a color video image signal (4:4:4 format) in which all the colorcomponents have the same sample number, such as an RGB signal, all thecolor components have a size of (L^(n), M^(n)), while in the case ofhandling a 4:2:0 format, a corresponding color difference component hasan encoding block size of (L^(n)/2, M^(n)/2). Hereafter, an encodingmode selectable for each block to be encoded B^(n) in the nthhierarchical layer is expressed as m(B^(n)).

In the case of a color video signal which consists of a plurality ofcolor components, the encoding mode m(B^(n)) can be formed in such a waythat an individual mode is used for each color component. Hereafter, anexplanation will be made by assuming that the encoding mode m(B^(n))indicates the one for the luminance component of each block to beencoded having a 4:2:0 format in a YUV signal unless otherwisespecified. The encoding mode m(B^(n)) can be one of one or more intraencoding modes (generically referred to as “INTRA”) or one or more interencoding modes (generically referred to as “INTER”), and the encodingcontrolling part 1 selects, as the encoding mode m(B^(n)), an encodingmode with the highest degree of encoding efficiency for each block to beencoded B^(n) from among all the encoding modes available in the picturecurrently being processed or a subset of these encoding modes, asmentioned above.

Each block to be encoded B^(n) is further divided into one or moreprediction units (partitions) by the block dividing part, as shown inFIG. 9. Hereafter, each partition belonging to each block to be encodedB^(n) is expressed as P_(i) ^(n) (i shows a partition number in the nthhierarchical layer). How the division of each block to be encoded B^(n)into partitions P_(i) ^(n) belonging to the block to be encoded B^(n) iscarried out is included as information in the encoding mode m(B^(n)).While the prediction process is carried out on each of all thepartitions P_(i) ^(n) according to the encoding mode m(B^(n)), anindividual prediction parameter can be selected for each partition P_(i)^(n).

The encoding controlling part 1 produces such a block division state asshown in, for example, FIGS. 10 a & 10 b for a block to be encodedhaving the maximum size, and then determines blocks to be encoded B^(n).Hatched portions shown in FIG. 10( a) show a distribution of partitionsinto which the block to be encoded having the maximum size is divided,and FIG. 10( b) shows a situation in which encoding modes m(B^(n)) arerespectively assigned to the partitions generated through thehierarchical layer division by using a quadtree graph. Each nodeenclosed by shown in FIG. 10( b) is a node (block to be encoded B^(n))to which an encoding mode m(B^(n)) is assigned.

When the encoding controlling part 1 selects an optimal encoding modem(B^(n)) for each partition P_(i) ^(n) of each block to be encodedB^(n), and the encoding mode m(B^(n)) is an intra encoding mode (stepST3), the selection switch 3 outputs the partition P_(i) ^(n) of theblock to be encoded B^(n), which is generated through the division bythe block dividing part 2, to the intra prediction part 4. In contrast,when the encoding mode m(B^(n)) is an inter encoding mode (step ST3),the selection switch outputs the partition P_(i) ^(n) of the block to beencoded B^(n), which is generated through the division by the blockdividing part 2, to the motion-compensated prediction part 5.

When receiving the partition P_(i) ^(n) of the block to be encoded B^(n)from the selection switch 3, the intra prediction part 4 carries out anintra prediction process on the partition P_(i) ^(n) of the block to beencoded B^(n) by using the intra prediction parameters corresponding tothe encoding mode m(B^(n)) selected by the encoding controlling part 1to generate an intra prediction image P_(i) ^(n) (step ST4). The intraprediction part 4 outputs the intra prediction image P_(i) ^(n) to thesubtracting part 6 and the adding part 9 after generating the intraprediction image P_(i) ^(n), while outputting the intra predictionparameters to the variable length encoding part 13 to enable the movingimage decoding device shown in FIG. 5 to generate the same intraprediction image P_(i) ^(n). Although the intra prediction process shownin this Embodiment 1 is not limited to the one according to an algorithmdetermined in the AVC/H.264 standards (ISO/IEC 14496-10), the intraprediction parameters need to include information required for themoving image encoding device and the moving image decoding device togenerate the completely same intra prediction image.

When receiving the partition P_(i) ^(n) of the block to be encoded B^(n)from the selection switch 3, and the encoding mode m(B^(n)) selected bythe encoding controlling part 1 is an inter encoding mode which is adirect mode, the motion-compensated prediction part 5 generates aspatial direct vector in the spatial direct mode from the motion vectorof an already-encoded block located in the vicinity of the partitionP_(i) ^(n) of the block to be encoded B^(n), and also generates atemporal direct vector in the temporal direct mode from the motionvector of an already-encoded picture which can be referred to by theblock to be encoded B^(n). The motion-compensated prediction part 5 thenselects one direct vector which provides a higher correlation betweenreference images from the spatial direct vector and the temporal directvector, and performs a motion-compensated prediction process on thepartition P_(i) ^(n) of the block to be encoded B^(n) by using thedirect vector selected thereby and the inter prediction parameterscorresponding to the encoding mode m(B^(n)) to generate a predictionimage (step ST5).

In contrast, when the encoding mode m(B^(n)) selected by the encodingcontrolling part 1 is an inter encoding mode other than direct modes,the motion-compensated prediction part 5 searches through the partitionP_(i) ^(n) of the block to be encoded B^(n) and a reference image storedin the motion-compensated prediction frame memory 12 for a motionvector, and carries out a motion-compensated prediction process on thepartition P_(i) ^(n) of the block to be encoded B^(n) by using themotion vector and the inter prediction parameters corresponding to theencoding mode m(B^(n)) to generate a prediction image (step ST5). Themotion-compensated prediction part 5 outputs the inter prediction imageP_(i) ^(n) to the subtracting part 6 and the adding part 9 aftergenerating the inter prediction image P_(i) ^(n), while outputting theinter prediction parameters to the variable length encoding part 13 toenable the moving image decoding device shown in FIG. 5 to generate thesame inter prediction image P_(i) ^(n). The inter prediction parametersused for the generation of the inter prediction image include:

-   -   Mode information in which the division of the block to be        encoded B^(n) into partitions is described;    -   The motion vector of each partition;    -   Reference image indication index information showing which        reference image is used for performing a prediction when the        motion-compensated prediction frame memory 12 stores a plurality        of reference images;    -   Index information showing which motion vector predicted value is        selected and used when there are a plurality of motion vector        predicted value candidates;    -   Index information showing which filter is selected and used when        there are a plurality of motion compensation interpolation        filters; and    -   Selection information showing which pixel accuracy is used when        the motion vector of the partition currently being processed can        show a plurality of degrees of pixel accuracy (half pixel, ¼        pixel, ⅛ pixel, etc.).        The inter prediction parameters are multiplexed into the        bitstream by the variable length encoding part 13 in order to        enable the moving image decoding device to generate the        completely same inter prediction image. The outline of the        process carried out by the motion-compensated prediction part 5        is as mentioned above, and the details of the process will be        mentioned below.

After the intra prediction part 4 or the motion-compensated predictionpart 5 generates a prediction image (an intra prediction image P_(i)^(n) or an inter prediction image P_(i) ^(n)), the subtracting part 6subtracts the prediction image (the intra prediction image P_(i) ^(n) orthe inter prediction image P_(i) ^(n)) generated by the intra predictionpart 4 or the motion-compensated prediction part 5 from the partitionP_(i) ^(n) of the block to be encoded B^(n), which is generated throughthe division by the block dividing part 2, to generate a differenceimage, and outputs a prediction difference signal e_(i) ^(n) showing thedifference image to the transformation/quantization part 7 (step ST6).

When receiving the prediction difference signal e_(i) ^(n) showing thedifference image from the subtracting part 6, thetransformation/quantization part 7 carries out a transforming process(e.g., a DCT (discrete cosine transform) or an orthogonal transformationprocess, such as a KL transform, in which bases are designed for aspecific learning sequence in advance) on the difference image in unitsof a block having the transformation block size included in theprediction difference encoding parameters outputted thereto from theencoding controlling part 1, and quantizes the transform coefficients ofthe difference image by using the quantization parameter included in theprediction difference encoding parameters and outputs the transformcoefficients quantized thereby to the inverse quantization/inversetransformation part 8 and the variable length encoding part 13 ascompressed data of the difference image (step ST7).

When receiving the compressed data of the difference image from thetransformation/quantization part 7, the inverse quantization/inversetransformation part 8 inverse-quantizes the compressed data of thedifference image by using the quantization parameter included in theprediction difference encoding parameters outputted thereto from theencoding controlling part 1, performs an inverse transformation process(e.g., an inverse DCT (inverse discrete cosine transform) or an inversetransformation process such as an inverse KL transform) on thecompressed data inverse-quantized thereby in units of a block having thetransformation block size included in the prediction difference encodingparameters, and outputs the compressed data on which the inversequantization/inverse transformation part performs the inversetransformation process as a local decoded prediction difference signale_(i) ^(n) hat (“̂” attached to an alphabetical letter is expressed byhat for reasons of the restrictions on electronic applications) (stepST8).

When receiving the local decoded prediction difference signal e_(i) ^(n)hat from the inverse quantization/inverse transformation part 8, theadding part 9 adds the local decoded prediction difference signal e_(i)^(n) hat and the prediction signal showing the prediction image (theintra prediction image P_(i) ^(n) or the inter prediction image P_(i)^(n)) generated by the intra prediction part 4 or the motion-compensatedprediction part 5 to generate a local decoded image which is a localdecoded partition image P_(i) ^(n) hat or a local decoded block to beencoded image which is a group of local decoded partition images (stepST9). After generating the local decoded image, the adding part 9 storesa local decoded image signal showing the local decoded image in thememory 10 for intra prediction and also outputs the local decoded imagesignal to the loop filter part 11.

The moving image encoding device repeatedly carries out the processes ofsteps ST3 to ST9 until the moving image encoding device completes theprocessing on all the blocks to be encoded B^(n) into which the inputtedimage is divided hierarchically, and, when completing the processing onall the blocks to be encoded B^(n), shifts to a process of step ST12(steps ST10 and ST11).

The variable length encoding part 13 entropy-encodes the compressed dataoutputted thereto from the transformation/quantization part 7, theencoding mode (including the information showing the state of thedivision into the blocks to be encoded) and the prediction differenceencoding parameters, which are outputted thereto from the encodingcontrolling part 1, and the intra prediction parameters outputtedthereto from the intra prediction part 4 or the inter predictionparameters outputted thereto from the motion-compensated prediction part5. The variable length encoding part 13 multiplexes encoded data whichare the encoded results of the entropy encoding of the compressed data,the encoding mode, the prediction difference encoding parameters, andthe intra prediction parameters or the inter prediction parameters togenerate a bitstream (step ST12).

When receiving the local decoded image signal from the adding part 9,the loop filter part 11 compensates for an encoding distortion includedin the local decoded image signal, and stores the local decoded imageshown by the local decoded image signal on which the loop filter partperforms the encoding distortion compensation in the motion-compensatedprediction frame memory 12 as a reference image (step ST13). The loopfilter part 11 can carry out the filtering process for each block to beencoded having the maximum size of the local decoded image signaloutputted thereto from the adding part 9 or for each block to beencoded. As an alternative, after the local decoded image signalcorresponding to all the macroblocks of one screen is outputted, theloop filter part can carry out the filtering process on all themacroblocks of the one screen at a time.

Next, the processing carried out by the motion-compensated predictionpart 5 will be explained in detail. When the encoding mode m(B^(n))selected by the encoding controlling part 1 is an inter mode other thandirect modes, the selection switch 21 of the motion-compensatedprediction part 5 outputs each of the partitions P_(i) ^(n) into whichthe block to be encoded B^(n) is divided by the block dividing part 2 tothe motion vector searching part 22. In contrast, when the encoding modem(B^(n)) is an inter mode which is a direct mode, the selection switchoutputs each of the partitions P_(i) ^(n) into which the block to beencoded B^(n) is divided by the block dividing part 2 to the directvector generating part 23. In this case, because the direct vectorgenerating part 23 does not use each of the partitions P_(i) ^(n) of theblock to be encoded B^(n) for the generation of a direct vector, thedirect vector generating part does not have to output each of thepartitions P_(i) ^(n) of the block to be encoded B^(n) to the directvector generating part 23 even though the encoding mode m(B^(n)) is aninter mode which is a direct mode.

When receiving each of the partitions P_(i) ^(n) of the block to beencoded B^(n) from the selection switch 21, the motion vector searchingpart 22 of the motion-compensated prediction part 5 searches for anoptimal motion vector in the inter mode while referring to the partitionP_(i) ^(n) and a reference image stored in the motion-compensatedprediction frame memory 12, and outputs the motion vector to the motioncompensation processing part 24. Because the process of searching for anoptimal motion vector in the inter mode is a known technique, a detailedexplanation of the process will be omitted hereafter.

When encoding mode m(B^(n)) is a direct mode, the direct vectorgenerating part 23 of the motion-compensated prediction part 5 generatesboth a spatial direct vector in the spatial direct mode and a temporaldirect vector in the temporal direct mode for each of the partitionsP_(i) ^(n) of the block to be encoded B^(n), and outputs either of thespatial direct vector and the temporal direct vector to the motioncompensation processing part 24 as a motion vector. Because theinformation showing the state of the division into the partitions P_(i)^(n) belonging to the block to be encoded B^(n) is included in theencoding mode m(B^(n)), as mentioned above, the direct vector generatingpart 23 can specify each of the partitions P_(i) ^(n) of the block to beencoded B^(n) by referring to the encoding mode m(B^(n)).

More specifically, the spatial direct vector generating part 31 of thedirect vector generating part 23 reads the motion vector of analready-encoded block located in the vicinity of each of the partitionsP_(i) ^(n) of the block to be encoded B^(n) from among the motionvectors of already-encoded blocks stored in the not-shown motion vectormemory or the not-shown internal memory to generate a spatial directvector in the spatial direct mode from the motion vector. Further, thetemporal direct vector generating part 32 of the direct vectorgenerating part 23 reads the motion vector of a block located spatiallyat the same position as each of the partitions P_(i) ^(n) of the blockto be encoded B^(n), which is the motion vector of an already-encodedpicture which can be referred to by the block to be encoded B^(n), fromamong the motion vectors of already-encoded blocks to generate atemporal direct vector in the temporal direct mode from the motionvector.

FIG. 11 is a schematic diagram showing a method of generating a motionvector (temporal direct vector) in the temporal direct mode. Forexample, a case in which a block MB1 in a picture B2 is the partitionP_(i) ^(n) which is the target to be encoded, and the block MB1 isencoded in the temporal direct mode is taken as an example. In thisexample, the temporal direct vector generating part uses the motionvector MV of a block MB2 which is the motion vector of a picture P3closest to the picture B2 among the already-encoded pictures locatedbackward with respect to the picture B2 on the time axis, and which isspatially located at the same position as the block MB 1. This motionvector MV refers to a picture P0, and motion vectors MVL and MVL1 whichare used when encoding the block MB1 are calculated according to thefollowing equation (3).

$\begin{matrix}{{{{MVL}\; 0} = {\frac{{T\; 2} - {T\; 0}}{{T\; 3} - {T\; 0}} \times M\; V}}{{{MVL}\; 1} = {\frac{{T\; 2} - {T\; 3}}{{T\; 3} - {T\; 0}} \times M\; V}}} & (3)\end{matrix}$

After calculating the motion vectors MVL0 and MVL1, the temporal directvector generating part 32 outputs the motion vectors MVL0 and MVL1 tothe direct vector determining part 33 as temporal direct vectors in thetemporal direct mode. Although as the method of generating a temporaldirect vector which the temporal direct vector generating part 32 uses,an H.264 method as shown in FIG. 11 can be used, this embodiment is notlimited to this method and another method can be alternatively used.

FIG. 12 is a schematic diagram showing the method of generating a motionvector (spatial direct vector) in the spatial direct mode. In FIG. 12,currentMB denotes the partition P_(i) ^(n) which is the block to beencoded. At this time, when the motion vector of an already-encodedblock A on a left side of the block to be encoded is expressed as MVa,the motion vector of an already-encoded block B on an upper side of theblock to be encoded is expressed as MVb, and the motion vector of analready-encoded block C on an upper right side of the block to beencoded is expressed as MVc, the spatial direct vector generating partcan calculate the motion vector MV of the block to be encoded bydetermining the median of these motion vectors MVa, MVb, and MVc, asshown by the following equation (4).

MV=median(MVa, MVb, MVc)   (4)

In the spatial direct mode, the spatial direct vector generating partdetermines the motion vector for each of the list 0 and the list 1. Inthis case, the spatial direct vector generating part can determine themotion vector for both of the lists by using the above-mentioned method.After calculating the motion vector MV for both the list 0 and the list1 in the above-mentioned way, the spatial direct vector generating part31 outputs the motion vector MV of the list 0 and that of the list 1 tothe direct vector determining part 33 as spatial direct vectors in thespatial direct mode. Although as the method of generating a spatialdirect vector which the spatial direct vector generating part 31 uses,an H.264 method as shown in FIG. 12 can be used, this embodiment is notlimited to this method and another method can be alternatively used.

For example, as shown in FIG. 13, the spatial direct vector generatingpart can select three motion vectors from a group of blocks A1 to An, agroup of blocks B1 to B^(n), and a group of blocks C, D, and E ascandidates for median prediction, respectively, to generate a spatialdirect vector. Further, in a case of ref_Idx in which the candidates forMV which are used for the generation of a spatial direct vector differfrom one another, the spatial direct vector generating part can carryout scaling according to the distance in the temporal direction, asshown in FIG. 14.

$\begin{matrix}{{scaled\_ MV} = {M\; V\frac{d({Xr})}{d({Yr})}}} & (5)\end{matrix}$

where scaled_MV denotes a scaled vector, MV denotes a motion vector yetto be scaled, and d(x) denotes the temporal distance to x. Further, Xrdenotes the reference image shown by the block to be encoded, and Yrdenotes the reference image show by each of block positions A to D whichis the target for scaling.

After the spatial direct vector generating part 31 generates spatialdirect vectors, the direct vector determining part 33 of the directvector generating part 23 calculates an evaluated value in the spatialdirect mode by using the spatial direct vectors. After the temporaldirect vector generating part 32 generates temporal direct vectors, thedirect vector determining part 33 calculates an evaluated value in thetemporal direct mode by using the temporal direct vectors. The directvector determining part 33 compares the evaluated value in the spatialdirect mode with the evaluated value in the temporal direct mode, andselects a direct vector in a direct mode from the spatial direct vectorand the temporal direct vector by using a determining part which will bementioned below, and outputs the direct vector to the motioncompensation processing part 24.

Hereafter, the processing carried out by the direct vector determiningpart 33 will be explained concretely. After the spatial direct vectorgeneration part 31 generates the spatial direct vectors MVL0 and MVL1,the motion compensation part 41 of the direct vector determining part 33generates a list 0 prediction image in the spatial direct mode by usingthe spatial direct vector MVL0, and also generates a list 1 predictionimage in the spatial direct mode by using the spatial direct vectorMVL1. FIG. 15 is an explanatory drawing showing an example of thecalculation of an evaluated value by using the degree of similaritybetween a forward prediction image and a backward prediction image. Inthe example shown in FIG. 15, the motion compensation part generates aforward prediction image f_(spatial) as the list 0 prediction image inthe spatial direct mode, and also generates a backward prediction imageg_(spatial) as the list 1 prediction image in the spatial direct mode.

After the temporal direct vector generating part 32 generates thetemporal direct vectors which are the motion vectors MV of the list 0and the list 1, the motion compensation part 41 further generates a list0 prediction image in the temporal direct mode by using the temporaldirect vector which is a forward motion vector MV, and also generates alist 1 prediction image in the temporal direct mode by using thetemporal direct vector which is a backward motion vector MV. In theexample shown in FIG. 15, the motion compensation part generates aforward prediction image f_(temporal) in the temporal direct mode as thelist 0 prediction image in the g_(temporal) direct mode, and alsogenerates a direct mode.

Although in this example the motion compensation part generates aforward prediction image as the list 0 prediction image by using areference image list 0 showing a reference image in a forward directionand also generates a backward prediction image as the list 1 predictionimage by using a reference image list 1 showing a reference image in abackward direction, the motion compensation part can alternativelygenerate a backward prediction image as the list 0 prediction image byusing a reference image list 0 showing a reference image in a backwarddirection and also generate a forward prediction image as the list 1prediction image by using a reference image list 1 showing a referenceimage in a forward direction. As an alternative, the motion compensationpart can generate forward prediction images as the list 0 predictionimage and the list 1 prediction image by using a reference image list 0showing a reference image in a forward direction and a reference imagelist 1 showing a reference image in a forward direction, respectively(this process will be mentioned below in detail).

After the motion compensation part generates the list 0 prediction imageand the list 1 prediction image in the spatial direct mode, thesimilarity calculating part 42 of the direct vector determining part 33calculates an evaluated value SAD_(spatial) in the spatial direct mode,as shown in the following equation (6). For the sake of simplicity, thelist 0 prediction image in the spatial direct mode is a forwardprediction image f_(spatial), and the list 1 prediction image in thespatial direct mode is a backward prediction image g_(spatial) inequation (6).

SAD_(spatial) =|f _(spatial) −g _(spatial)|  (6)

Further, after the motion compensation part generates the list 0prediction image and the list 1 prediction image in the temporal directmode, the similarity calculating part 42 calculates an evaluated valueSAD_(temporal) in the temporal direct mode, as shown in the followingequation (7). For the sake of simplicity, the list 0 prediction image inthe temporal direct mode is a forward prediction image f_(temporal), andthe list 1 prediction image in the spatial direct mode is a backwardprediction image g_(temporal) in equation (7).

SAD_(temporal) −|f _(temporal) −g _(temporal)|  (7)

The larger the difference between the forward prediction image and thebackward prediction image is, the lower the degree of similarity betweenthe two images is (the evaluated value SAD showing the sum of absolutedifferences between the two images becomes large), and the lower thetemporal correlation between them is. In contrast with this, the smallerthe difference between the forward prediction image and the backwardprediction image is, the higher the degree of similarity between the twoimages is (the evaluated value SAD showing the sum of absolutedifferences between the two images becomes small), and the higher thetemporal correlation between them is. Further, an image which ispredicted from a direct vector must be an image which is similar to theblock to be encoded. Therefore, when prediction images are generated byusing two vectors, respectively, the images which are predictedrespectively from the vectors are expected to resemble the block to beencoded, and this means that there is a high correlation between the tworeference images. Therefore, by selecting a direct vector having asmaller evaluated value SAD from the spatial direct vector and thetemporal direct vector, the direct vector determining part can select amode which provides a high correlation between reference images, andhence can improve the accuracy of the direct mode.

After the similarity calculating part 42 calculates both the evaluatedvalue SAD_(spatial) in the spatial direct mode and the evaluated valueSAD_(temporal) in the temporal direct mode, the direct vector selectingpart 43 of the direct vector determining part 33 compares the degree ofsimilarity between the forward prediction image f_(spatial) and thebackward prediction image g_(spatial) in the spatial direct mode withthe degree of similarity between the forward prediction imagef_(temporal) and the backward prediction image g_(temporal) in thetemporal direct mode by comparing the evaluated value SAD_(spatial) withthe evaluated value SAD_(temporal).

When the degree of similarity between the forward prediction imagef_(spatial) and the backward prediction image g in the spatial directmode is equal to or higher than the degree of similarity between theforward prediction image f_(temporal) and the backward prediction imageg_(temporal) in the temporal direct mode (SAD_(spatial)SAD_(temporal))the direct vector selecting part 43 selects the spatial direct vectorgenerated by the spatial direct vector generating part 31, and outputsthe spatial direct vector to the motion compensation processing part 24as a motion vector. In contrast, when the degree of similarity betweenthe forward prediction image f_(temporal) and the backward predictionimage g_(temporal) in the temporal direct mode is higher than the degreeof similarity between the forward prediction image f_(spatial) and thebackward prediction image g_(spatial) in the spatial direct mode(SAD_(spatial)>SAD_(temporal)), the direct vector selecting part selectsthe temporal direct vector generated by the temporal direct vectorgenerating part 32, and outputs the temporal direct vector to the motioncompensation processing part 24 as a motion vector.

When the encoding mode m(B^(n)) is not a direct mode, and the motioncompensation processing part 24 receives the motion vector from themotion vector searching part 22, the motion compensation processing part24 carries out a motion-compensated prediction process on the basis ofthe inter prediction parameters outputted thereto from the encodingcontrolling part 1 by using both the motion vector and one frame ofreference image stored in the motion-compensated prediction frame memory12 to generate a prediction image. In contrast, when the encoding modem(B^(n)) is a direct mode and the motion compensation processing part 24receives the motion vector (i.e., the direct vector selected by thedirect vector selection part 43) from the direct vector generating part23, the motion compensation processing part 24 carries out amotion-compensated prediction process on the basis of the interprediction parameters outputted thereto from the encoding controllingpart 1 by using both the motion vector and one frame of reference imagestored in motion-compensated prediction frame memory 12 to generate aprediction image. Because the motion-compensated prediction processcarried out by the motion compensation processing part 24 is a knowntechnique, a detailed explanation of the motion-compensated predictionprocess will be omitted hereafter.

Although the example in which the similarity calculating part 42calculates the evaluated value SAD which is the sum of absolutedifferences between the two images both in the temporal direct mode andin the spatial direct mode and the direct vector selecting part 43compares the evaluated value SAD in the temporal direct mode with thatin the spatial direct mode is shown, the similarity calculating part 42can alternatively calculate the sum of the squares of differences SSEbetween the forward prediction image and the backward prediction imageboth in the temporal direct mode and in the spatial direct mode asevaluated values, and the direct vector selecting part 43 can comparethe sum of the squares of differences SSE in the temporal direct modewith that in the spatial direct mode. While the use of SSE increases theamount of information to be processed, the degree of similarity can becalculated more correctly.

Next, the processing carried out by the image decoding device shown inFIG. 5 will be explained. When receiving the bitstream outputted theretofrom the image encoding device of FIG. 1, the variable length decodingpart 51 carries out a variable length decoding process on the bitstreamto decode the frame size in units of a sequence which consists of one ormore frames of pictures or in units of a picture (step ST21 of FIG. 8).The variable length decoding part 51 determines a maximum size of eachof coding blocks which is a unit to be processed at a time when amotion-compensated prediction process (inter-frame prediction process)or an intra prediction process (intra-frame prediction process) iscarried out according to the same procedure as that which the encodingcontrolling part 1 shown in FIG. 1 uses, and also determines an upperlimit on the number of hierarchical layers in a hierarchy in which eachof the coding blocks having the maximum size is hierarchically dividedinto blocks (step ST22). For example, when the maximum size of each ofcoding blocks is determined according to the resolution of the inputtedimage in the image encoding device, the variable length decoding partdetermines the maximum size of each of the coding blocks on the basis ofthe frame size which the variable length decoding part has decodedpreviously. When information showing both the maximum size of each ofthe coding blocks and the upper limit on the number of hierarchicallayers is multiplexed into the bitstream, the variable length decodingpart refers to the information which is acquired by decoding thebitstream.

Because the information showing the state of the division of each of thecoding blocks B⁰ having the maximum size is included in the encodingmode m(B⁰) of the coding block B⁰ having the maximum size which ismultiplexed into the bitstream, the variable length decoding part 51specifies each of the coding blocks B^(n) into which the image isdivided hierarchically by decoding the bitstream to acquire the encodingmode m(B⁰) of the coding block B⁰ having the maximum size which ismultiplexed into the bitstream (step ST23). After specifying each of thecoding blocks B^(n), the variable length decoding part 51 decodes thebitstream to acquire the encoding mode m(B^(n)) of the coding blockB^(n) to specify each partition P_(i) ^(n) belonging to the coding blockB^(n) on the basis of the information about the partition P_(i) ^(n)belonging to the encoding mode m(B^(n)). After specifying each partitionP_(i) ^(n) belonging to the coding block B^(n), the variable lengthdecoding part 51 decodes the encoded data to acquire the compresseddata, the encoding mode, the prediction difference encoding parameters,and the intra prediction parameters/inter prediction parameters for eachpartition P_(i) ^(n) (step ST24).

When the encoding mode m(B^(n)) of the partition P_(i) ^(n) belonging tothe coding block B^(n), which is specified by the variable lengthdecoding part 51, is an intra encoding mode (step ST25), the selectionswitch 52 outputs the intra prediction parameters outputted thereto fromthe variable length decoding part 51 to the intra prediction part 53. Incontrast, when the encoding mode m(B^(n)) of the partition P_(i) ^(n) isan inter encoding mode (step ST25), the selection switch outputs theinter prediction parameters outputted thereto from the variable lengthdecoding part 51 to the motion-compensated prediction part 54. Whenreceiving the intra prediction parameters from the selection switch 52,the intra prediction part 53 carries out an intra prediction process onthe partition P_(i) ^(n) of the coding block B^(n) by using the intraprediction parameters to generate an intra prediction image P_(i) ^(n)(step ST26).

When receiving the inter prediction parameters from the selection switch52 and the encoding mode m(B^(n)) outputted thereto from the variablelength decoding part 51 is an inter encoding mode which is a directmode, the motion-compensated prediction part 54 generates a spatialdirect vector in the spatial direct mode and a temporal direct vector inthe temporal direct mode, like the motion-compensated prediction part 5shown in FIG. 1. After generating a spatial direct vector in the spatialdirect mode and a temporal direct vector in the temporal direct mode,the motion-compensated prediction part 54 selects one direct vectorwhich provides a higher correlation between reference images from thespatial direct vector and the temporal direct vector, like themotion-compensated prediction part 5 shown in FIG. 1, and carries out amotion-compensated prediction process on the partition P_(i) ^(n) of thecoding block B^(n) by using the direct vector selected thereby and theinter prediction parameters to generate an inter prediction image P_(i)^(n) (step ST27).

In contrast, when the encoding mode m(B^(n)) outputted thereto from thevariable length decoding part 51 is an inter encoding modes other thandirect modes, the motion compensation processing part 63 of themotion-compensated prediction part 54 carries out a motion-compensatedprediction process on the partition P_(i) ^(n) of the coding block B^(n)by using the motion vector included in the inter prediction parametersoutputted thereto from the selection switch 52 to generate an interprediction image P_(i) ^(n) (step ST27).

The inverse quantization/inverse transformation part 55inverse-quantizes the compressed data associated with the coding block,which are outputted thereto from the variable length decoding part 51,by using the quantization parameter included in the predictiondifference encoding parameters outputted thereto from the variablelength decoding part 51, and performs an inverse transformation process(e.g., an inverse DCT (inverse discrete cosine transform) or an inversetransformation process such as an inverse KL transform) on thecompressed data inverse-quantized thereby in units of a block having thetransformation block size included in the prediction difference encodingparameters, and outputs the compressed data on which the inversequantization/inverse transformation part performs the inversetransformation process to the adding part 56 as a decoded predictiondifference signal (signal showing a pre-compressed difference image)(step ST28).

When receiving the decoded prediction difference signal from the inversequantization/inverse transformation part 55, the adding part 56generates a decoded image by adding the decoded prediction differencesignal and the prediction signal showing the prediction image generatedby the intra prediction part 53 or the motion-compensated predictionpart 54 and stores the decoded image signal showing the decoded image inthe memory 57 for intra prediction, and also outputs the decoded imagesignal to the loop filter part 58 (step ST29).

The moving image decoding device repeatedly carries out the processes ofsteps ST23 to ST29 until the moving image decoding device completes theprocessing on all the coding blocks B^(n) into which the image isdivided hierarchically (step ST30). When receiving the decoded imagesignal from the adding part 56, the loop filter part 58 compensates foran encoding distortion included in the decoded image signal, and storesthe decoded image shown by the decoded image signal on which the loopfilter part performs the encoding distortion compensation in themotion-compensated prediction frame memory 59 as a reference image (stepST31). The loop filter part 58 can carry out the filtering process foreach coding block having the maximum size of the local decoded imagesignal outputted thereto from the adding part 56 or each coding block.As an alternative, after the local decoded image signal corresponding toall the macroblocks of one screen is outputted, the loop filter part cancarry out the filtering process on all the macroblocks of the one screenat a time.

As can be seen from the above description, the moving image encodingdevice in accordance with this Embodiment 1 is constructed in such a waythat the moving image encoding device includes: the encoding controllingpart 1 for determining a maximum size of each of blocks to be encodedwhich is a unit to be processed at a time when a prediction process iscarried out, and also determining a hierarchy number upper limit on thenumber of hierarchical layers in a hierarchy in which each of the blocksto be encoded having the maximum size is hierarchically divided intoblocks, and for selecting an encoding mode suitable for each of theblocks to be encoded into which each block to be encoded having themaximum size is divided hierarchically from one or more availableencoding modes; and the block dividing part 2 for dividing an inputtedimage into blocks to be encoded each having the maximum size determinedby the encoding controlling part 1, and also dividing each of the blocksto be encoded hierarchically until its hierarchy number reaches thehierarchy number upper limit determined by the encoding controlling part1, and, when an inter encoding mode which is a direct mode is selectedby the encoding controlling part 1 as an encoding mode suitable for oneof the blocks to be encoded into which the inputted image is divided bythe block dividing part 2, the motion-compensated prediction part 5generates a spatial direct vector in a spatial direct mode from themotion vector of an already-encoded block located in the vicinity of theblock to be encoded and also generates a temporal direct vector in atemporal direct mode from the motion vector of an already-encodedpicture which can be referred to by the block to be encoded, selects onedirect vector which provides a higher correlation between referenceimages from the spatial direct vector and the temporal direct vector,and carries out a motion-compensated prediction process on the block tobe encoded by using the direct vector to generate a prediction image.Therefore, there is provided an advantage of being able to select anoptimal direct mode for each predetermined block unit, and reduce thecode amount.

Further, the moving image decoding device in accordance with thisEmbodiment 1 is constructed in such a way that the moving image decodingdevice includes the variable length decoding part 51 forvariable-length-decoding the encoded data to acquire the compressed dataand the encoding mode associated with each of coding blocks into whichan image is hierarchically divided from the encoded data multiplexedinto the bitstream, and, when the encoding mode associated with a codingblock variable-length-decoded by the variable length decoding part 51 isan inter encoding mode which is a direct mode, the motion-compensatedprediction part 54 generates a spatial direct vector in the spatialdirect mode from the motion vector of an already-decoded block locatedin the vicinity of the coding block and also generates a temporal directvector in the temporal direct mode from the motion vector of analready-decoded picture which can be referred to by the coding block,selects one direct vector which provides a higher correlation betweenreference images from the spatial direct vector and the temporal directvector, and carries out a motion-compensated prediction process on thecoding block by using the direct vector to generate a prediction image.Therefore, there is provided an advantage of making it possible for themoving image decoding device to decode the encoded data which enable theselection of an optimal direct mode for each fixed block unit.

Embodiment 2

In above-mentioned Embodiment 1, the example in which each of themotion-compensated prediction parts 5 and 54 (concretely, the similaritycalculating part 42) calculates the degree of similarity between aforward prediction image f_(spatial) and a backward prediction imageg_(spatial) in the spatial direct mode as an evaluated valueSAD_(spatial) in the spatial direct mode while calculating the degree ofsimilarity between a forward prediction image f_(temporal) and abackward prediction image g_(temporal) in the temporal direct mode as anevaluated value SAD_(temporal) in the temporal direct mode is shown,each of the motion-compensated prediction parts can alternativelycalculate a variance o (spatial) of the motion vectors ofalready-encoded blocks (decoded blocks) located in the vicinity of ablock to be encoded B^(n) as an evaluated value in the spatial directmode while calculating a variance o (temporal) of the motion vectors ofalready-encoded blocks (decoded blocks) located in the vicinity of ablock located spatially at the same position as the block to be encodedB^(n) in an encoded picture (decoded picture) which can be referred toby the block to be encoded B^(n) as an evaluated value in the temporaldirect mode. This embodiment can provide the same advantages as thoseprovided by above-mentioned Embodiment 1.

More specifically, the similarity calculating part 42 calculates avariance σ (spatial) of the motion vectors of already-encoded blocks(decoded blocks) located in the vicinity of the block to be encodedB^(n) as the evaluated value SAD_(spatial) in the spatial direct mode(refer to the following equation (8)), as shown in FIG. 16( a), insteadof calculating the degree of similarity between the forward predictionimage f_(spatial) and the backward prediction image g_(spatial) in thespatial direct mode. Further, the similarity calculating part 42calculates a variance a (temporal) of the motion vectors ofalready-encoded blocks (decoded blocks) located in the vicinity of ablock located spatially at the same position as the block to be encodedB^(n) in an encoded picture (decoded picture) which can be referred toby the block to be encoded B^(n) as the evaluated value SAD_(temporal)in the temporal direct mode (refer to the following equation (8)), asshown in FIG. 16( b), instead of calculating the degree of similaritybetween the forward prediction image f_(temporal) and the backwardprediction image g_(temporal) in the temporal direct mode.

$\begin{matrix}{{{{\sigma (m)} = {\frac{1}{N}{\sum\limits_{i \in R}( {{MV}_{m,i} - {\overset{\_}{MV}}_{m}} )^{2}}}},{N = 4}}{{where}\mspace{14mu} {MV}_{m,i}\mspace{14mu} {is}\mspace{14mu} {the}\mspace{14mu} {motion}\mspace{14mu} {vector}\mspace{14mu} {of}\mspace{14mu} {an}\mspace{14mu} {adjacent}}\; \text{}{{block},\; {{and}\mspace{14mu} {\overset{\_}{MV}}_{m}\mspace{14mu} {is}\mspace{14mu} {the}\mspace{14mu} {average}\mspace{14mu} {of}\mspace{14mu} {the}\mspace{14mu} {motion}\mspace{14mu} {vectors}}}{{of}\mspace{14mu} {adjacent}\mspace{14mu} {{blocks}.}}} & (8)\end{matrix}$

Further, m is a symbol showing spatial or temporal.

The direct vector selecting part 43 compares the variance σ (temporal)of the motion vectors with the variance σ (spatial) of the motionvectors, and, when the variance σ (temporal) of the motion vectors isequal to or larger than the variance σ (spatial) of the motion vectors,determines that the reliability of the motion vector in the spatialdirect mode (spatial direct vector) is low, and selects the motionvector in the temporal direct mode (temporal direct vector). Incontrast, when the variance σ (spatial) of the motion vectors is largerthan the variance σ (temporal) of the motion vectors, the direct vectorselecting part determines that the reliability of the motion vector inthe temporal direct mode (temporal direct vector) is low, and selectsthe motion vector in the spatial direct mode (spatial direct vector).

Although the example in which each of the motion-compensated predictionparts generates both the temporal direct vector and the spatial directvector and selects either of the direct vectors is shown inabove-mentioned Embodiment 1, each of the motion-compensated predictionparts can add another vector, as a candidate vector, in addition to thetemporal direct vector and the spatial direct vector, and select onedirect vector from these candidate vectors. For example, each of themotion-compensated prediction parts can add spatial vectors MV_A, MV_B,and MV_C, and temporal vectors MV_(—)1 to MV_(—)8 as shown in FIG. 17 tothe candidate vectors, and select one direct vector from these spatialvectors and temporal vectors. Further, as shown in FIG. 18, each of themotion-compensated prediction parts can generate one vector from aplurality of already-encoded vectors, and add the vector to thecandidate vectors. While such an increase in the number of candidatevectors increases the amount of information to be processed, theprecision of the direct vector can be improved and hence the encodingefficiency can be improved.

Although no mention has been made particularly in above-mentionedEmbodiment 1, the candidates for the direct vector can be determined ona per slice basis. Information showing which vectors should be selectedas candidates is multiplexed into each slice header. For example, therecan be considered a method of, because the effect of a temporal vectoris low in a video which is acquired by panning a camera, removingtemporal vectors from the selection candidates for such a video, and,because the effect of a spatial vector is large in a video which isacquired by a fixed camera, adding spatial vectors to the selectioncandidates for such a video.

While the larger the number of candidate vectors is, the nearer to theoriginal image a prediction image can be generated, a balance betweenthe amount of information to be processed and the encoding efficiencycan be achieved by determining the candidates in consideration of thelocality of the video, such as excluding ineffective vectors from thecandidates in advance, in order to prevent the amount of information tobe processed from greatly increasing due to the increase in the numberof candidate vectors. Switching a vector between a candidate and anon-candidate is achieved by using, for example, a method of providingan ON/OFF flag for each vector, and defining only a vector whose flag isset to ON as a candidate. A motion vector which can be a selectioncandidate can be switched between a candidate and a non-candidate byusing each slice header or each header in an upper layer, such as eachsequence header or each picture header. Further, one or more sets ofmotion vectors each of which can be a selection candidate can beprepared, and an index showing each of the candidate sets can beencoded.

Further, a vector can be switched between a candidate and anon-candidate for each macroblock or each block to be encoded. Switchinga vector between a candidate and a non-candidate for each macroblock oreach block to be encoded can provide the macroblock or block to beencoded with locality, and provides an advantage of improving theencoding efficiency. Further, the selection candidates can be determineduniquely for each partition block size. Because the spatial correlationgenerally becomes weak as the block size becomes small, it can beexpected that the predictive accuracy of a vector determined through amedian prediction gets worse. To solve this problem, by removing amotion vector determined through a median prediction from thecandidates, for example, the amount of information to be processed canbe reduced without lowering the encoding efficiency.

Although the explanation is made in above-mentioned Embodiment 1 byassuming the case in which both a temporal direct vector and a spatialdirect vector exist, there is a case in which no motion vector existswhen an intra encoding process is carried out on the block to be encodedB^(n). In this case, there can be considered a method of setting a zerovector as a motion vector, a method of not including any motion vectorin the candidates, and so on. While the encoding efficiency can beimproved because the candidates increase in number when a zero vector isset as a motion vector, the amount of information to be processedincreases. When no motion vector is included in the candidates fordirect vector, the amount of information to be processed can be reduced.

Although the example of generating a direct vector is shown inabove-mentioned Embodiment 1, the direct vector can be used as apredicted vector which is used for encoding of a normal motion vector.While the amount of information to be processed increases when thedirect vector is used as a predicted vector, the encoding efficiency canbe improved because the accuracy of the prediction increases.

Although the example of calculating an evaluated value SAD from acombination of an image located backward of the block to be encodedB^(n) in time and an image located forward of the block to be encodedB^(n) in time is shown in above-mentioned Embodiment 1 (refer to FIG.15), an evaluated value SAD can be alternatively calculated from acombination of only images located backward of the block to be encodedB^(n) in time, as shown in FIG. 19. As an alternative, an evaluatedvalue SAD can be calculated from a combination of only images locatedforward of the block to be encoded B^(n) in time. In this case, temporalvectors are expressed by the following equations (9) and (10).

$\begin{matrix}{{\hat{v}}_{0} = {\frac{d_{0}}{d_{col}}v_{col}}} & (9) \\{{\hat{v}}_{1} = {\frac{d_{1}}{d_{col}}v_{col}}} & (10)\end{matrix}$

where {circumflex over (v)}₀ is the vector of list 0, and {circumflexover (v)}₁ is the vector of list 1.

In the above equations, d denotes a temporal distance, d_(o) denotes thetemporal distance of a list 0 reference image, and d₁ denotes thetemporal distance of a list 0 reference image. Further, v_(col) andd_(col) denote the vector of a block spatially located at the sameposition in the reference image as the block to be encoded, and thetemporal distance of the reference image shown by the vector,respectively.

Even in a case in which the two reference image lists indicate the samereference image, the same method as that shown in FIG. 19 can be appliedwhen each of the lists has two or more reference images.

Although the case in which each of the two reference image lists has twoor more reference images is assumed in above-mentioned Embodiment 1,there can be considered a case in which only one reference image isincluded in each of the two reference image lists. In this case, whenthe same reference image is set to the two reference image lists, therecan be a case in which the determination can be carried out by usingonly a spatial vector without using any temporal vector. When differentreference images are set to the two reference image lists, respectively,the determination can be handled by using the above-mentioned method.

Although a prediction process from two directions is assumed to becarried out in above-mentioned Embodiment 1, a prediction process onlyin a single direction can be alternatively carried out. When aprediction from a vector in one direction is carried out, informationshowing which vector is used is encoded and transmitted. As a result, aproblem, such as occlusion, can be dealt with, and a contribution to animprovement in the predictive accuracy can be made.

Although it is assumed in a direct mode shown in above-mentionedEmbodiment 1 that a prediction using two vectors is carried out, thenumber of vectors can be three or more. In this case, for example, therecan be considered a method of generating a prediction image by using allvector candidates each of which provides an evaluated value SAD equal toor smaller than a threshold Th, among a plurality of vector candidates.Further, a number of reference image lists whose number is equal to thenumber of vectors can be stored. Further, instead of using allcandidates each of which provides an evaluated value SAD equal to orsmaller than the threshold Th, a maximum of the number of vectors whichare used can be preset to each slice header or the like, and aprediction image can be generated by using the maximum number of vectorseach of which provides a smaller evaluated value. It is generally knownthat the performance is further improved with increase in the number ofreference images used for the generation of a prediction image.Therefore, while the amount of information to be processed increases, ancontribution to an improvement in the encoding efficiency can be made.

A vector is determined from an evaluation between reference images inabove-mentioned Embodiment 1. This evaluation can be carried out from acomparison between an already-encoded image which is spatially adjacentto the block to be encoded and a reference image. In this case, therecan be considered a method of carrying out the evaluation by using suchan L-shaped image as shown in FIG. 20. Further, when an already-encodedimage which is spatially adjacent to the block to be encoded is used,there is a possibility that the already-encoded image is not in time forthe comparison because of pipeline processing. In this case, there canbe considered a method of using the prediction image instead of thealready-encoded image.

Although the example in which the size of the block to be encoded B^(n)is L^(n)=M^(n) as shown in FIG. 9 is shown in above-mentioned Embodiment1, the size of the block to be encoded B^(n) can be L^(n)≠M^(n). Forexample, there can be considered a case in which the size of the blockto be encoded B^(n) is L^(n)=kM^(n) as shown in FIG. 21. In this case,(L^(n+1), M^(n+1)) becomes equal to (L^(n), M^(n)) in the next division,and subsequent divisions can be carried out in the same way as thoseshown in FIG. 9 or in such a way that (L^(n+1), M^(n+1)) becomes equalto (L^(n)/2, M^(n)/2) (refer to FIG. 22). As an alternative, one of adividing process shown in FIG. 21 and that shown in FIG. 22 can beselected as shown in FIG. 23. In the case in which one of the dividingprocess shown in FIG. 21 and that shown in FIG. 22 can be selected, aflag showing which division process is selected is encoded. Because thiscase can be implemented by using a method of connecting blocks eachconsists of 16×16 elements to each other in a horizontal direction, suchas H.264 disclosed by nonpatent reference 1, the compatibility with theexisting method can be maintained. Although the case in which the sizeof the block to be encoded B^(n) is L^(n)=kMn is shown in theabove-mentioned explanation, it needless to say that divisions can becarried out on the same principle even if blocks are connected to eachother in a vertical direction, like in a case of kL^(n)=Mn.

Although the transformation/quantization part 7 and the inversequantization/inverse transformation parts 8 and 55 carry outtransformation processes (inverse transformation processes) in units ofa block having the transformation block size included in the predictiondifference encoding parameters in above-mentioned Embodiment 1, eachtransformation block size unit can be determined uniquely by atransformation process part, or can be formed to have a hierarchicalstructure as shown in FIG. 24. In this case, a flag showing whether ornot a division is carried out for each hierarchical layer is encoded.The above-mentioned division can be carried out for each partition oreach block to be encoded. Although the above-mentioned transformation isassumed to be carried out in units of a square block, the transformationcan be alternatively carried out in units of a quadrangular block suchas a rectangular block.

Embodiment 3

Although the example in which each of the direct vector generating parts23 and 62 of the motion-compensated prediction parts 5 and 54 generatesboth a spatial direct vector and a temporal direct vector is shown inabove-mentioned Embodiment 1, each of the direct vector generating partscan alternatively determine an initial search point when generating botha spatial direct vector and a temporal direct vector, and search throughthe vicinity of the initial search point to determine a direct vector.

FIG. 25 is a block diagram showing a motion-compensated prediction part5 of a moving image encoding device in accordance with Embodiment 3 ofthe present invention. In the figure, because the same referencenumerals as those shown in FIG. 2 denote the same components or likecomponents, the explanation of the components will be omitted hereafter.A direct vector generating part 25 carries out a process of generatingboth a spatial direct vector and a temporal direct vector.

FIG. 26 is a block diagram showing the direct vector generation part 25which constructs the motion-compensated prediction part 5. Referring toFIG. 26, an initial vector generating part 34 carries out a process ofgenerating an initial vector from the motion vector of analready-encoded block. A motion vector searching part 35 carries out aprocess of searching through the vicinity of an initial search pointshown by the initial vector generated by the initial vector generatingpart 34 to determine a direct vector.

FIG. 27 is a block diagram showing the initial vector generating part 34which constructs the direct vector generating part 25. Referring to FIG.27, a spatial vector generating part 71 carries out a process ofgenerating a spatial vector from the motion vector of an already-encodedblock by using, for example, the same method as that which the spatialdirect vector generating part 31 shown in FIG. 3 uses. A temporal vectorgenerating part 72 carries out a process of generating a temporal vectorfrom the motion vector of an already-encoded block by using, forexample, the same method as that which the temporal direct vectorgenerating part 32 shown in FIG. 3 uses. An initial vector determiningpart 73 carries out a process of selecting either of the spatial vectorgenerated by the spatial vector generating part 71 and the temporalvector generated by the temporal vector generating part 72 as an initialvector.

FIG. 28 is a block diagram showing the initial vector determining part73 which constructs the initial vector generating part 34. Referring toFIG. 28, a motion compensation part 81 carries out a process ofgenerating a list 0 prediction image in a spatial direct mode, a list 1prediction image in the spatial direct mode, a list 0 prediction imagein a temporal direct mode, and a list 1 prediction image in the temporaldirect mode by using the same method as that which the motioncompensation part 41 shown in FIG. 4 uses.

A similarity calculating part 82 carries out a process of calculatingthe degree of similarity between the list 0 prediction image and thelist 1 prediction image in the spatial direct mode as a spatialevaluated value and also calculating the degree of similarity betweenthe list 0 prediction image and the list 1 prediction image in thetemporal direct mode as a temporal evaluated value by using the samemethod as that which the similarity calculating part 42 shown in FIG. 4.An initial vector determining part 83 carries out a process of making acomparison between the spatial evaluated value and the temporalevaluated value which are calculated by the similarity calculating part82 to select the spatial vector or the temporal vector according to thecomparison result.

FIG. 29 is a block diagram showing a motion-compensated prediction part54 of a moving image decoding device in accordance with Embodiment 3 ofthe present invention. In the figure, because the same referencenumerals as those shown in FIG. 6 denote the same components or likecomponents, the explanation of the components will be omitted hereafter.A direct vector generating part 64 carries out a process of generatingboth a spatial direct vector and a temporal direct vector. The internalstructure of the direct vector generating part 64 is the same as thedirect vector generating part 25 shown in FIG. 25.

Next, the operation of the moving image encoding device and theoperation of the moving image decoding device will be explained. Becausethe moving image encoding device and the moving image decoding deviceaccording to this embodiment have the same structures as those accordingto above-mentioned Embodiment 1, with the exception that the directvector generating parts 23 and 62 of the motion-compensated predictionparts 5 and 54 according to above-mentioned Embodiment 1 are replaced bythe direct vector generating parts 25 and 64, as compared withabove-mentioned Embodiment 1, only processes carried out by each of thedirect vector generating parts 25 and 64 will be explained hereafter.Because the process carried out by the direct vector generating part 25is the same as that carried out by the direct vector generating part 64,the process carried out by the direct vector generating part 25 will beexplained hereafter.

The initial vector generating part 34 of the direct vector generatingpart 25 generates an initial vector MV_first from the motion vector ofan already-encoded block. More specifically, the spatial vectorgenerating part 71 of the initial vector generating part 34 generates aspatial vector from the motion vector of an already-encoded block byusing, for example, the same method as that which the spatial directvector generating part 31 shown in FIG. 3 uses. As an alternative, thespatial vector generating part can generate a spatial vector by usinganother method. The temporal vector generating part 72 of the initialvector generating part 34 generates a temporal vector from the motionvector of an already-encoded block by using, for example, the samemethod as that which the temporal direct vector generating part 32 shownin FIG. 3 uses. As an alternative, the temporal vector generating partcan generate a temporal vector by using another method.

After the spatial vector generating part 71 generates a spatial vectorand the temporal vector generating part 72 generates a temporal vector,the initial vector determining part 73 of the initial vector generatingpart 34 selects one vector as an initial vector MV_first from thespatial vector and the temporal vector. More specifically, the motioncompensation part 81 of the initial vector determining part 73 generatesa list 0 prediction image in the spatial direct mode, a list 1prediction image in the spatial direct mode, a list 0 prediction imagein the temporal direct mode, and a list 1 prediction image in thetemporal direct mode by using the same method as that which the motioncompensation part 41 shown in FIG. 4 uses.

The similarity calculating part 82 of the initial vector determiningpart 73 calculates the degree of similarity between the list 0prediction image and the list 1 prediction image in the spatial directmode as a spatial evaluated value, and also calculates the degree ofsimilarity between the list 0 prediction image and the list 1 predictionimage in the temporal direct mode as a temporal evaluated value by usingthe same method as that which the similarity calculating part 42 shownin FIG. 4 uses. The initial vector determining part 83 of the initialvector determining part 73 refers to the result of the comparisonbetween the spatial evaluated value and the temporal evaluated valuewhich are calculated by the similarity calculating part 82, and selectsone vector which provides a higher degree of similarity betweenprediction images from the spatial vector and the temporal vector.

After the initial vector generating part 34 generates the initial vectorMV_first, the motion vector searching part 35 of the direct vectorgeneration part 25 searches through a range of ±n centered at an initialsearch point (block) shown by the initial vector MV_first, as shown inFIG. 30, to determine a direct vector. The motion vector searching partcan carry out an evaluation at the time of the search by carrying out,for example, the same process as that performed by the similaritycalculating part 82 shown in FIG. 28. In this case, when the positionshown by the initial vector is expressed as v, the motion vectorsearching part calculates an evaluated value SAD at the time of thesearch, as shown in the following equation (11).

SAD=|f(v ₁ −x)−g(v ₂ +x)|  (11)

In this case, the search range of n can be fixed or can be determinedfor each header in an upper layer such as each slice header. Further,although the range (search range) of the search point is assumed to be asquare, the range can be alternatively a rectangle or a quadrangle suchas a lozenge.

After calculating the evaluated value SAD at the time of the search, themotion vector searching part 35 outputs a motion vector in the searchrange which provides the smallest evaluated value SAD to the motioncompensation processing part 24 as a direct vector.

Although the example in which each of the motion-compensated predictionparts generates both a temporal direct vector and a spatial directvector and selects either of the direct vectors is shown inabove-mentioned Embodiment 3, each of the motion-compensated predictionparts can add another vector, as a candidate vector, in addition to thetemporal direct vector and the spatial direct vector, and select adirect vector from these candidate vectors. For example, each of themotion-compensated prediction parts can add spatial vectors MV_A, MV_B,and MV_C, and temporal vectors MV_(—)1 to MV_(—)8 as shown in FIG. 17 tothe candidate vectors, and select a direct vector from these spatialvectors and temporal vectors. Further, each of the motion-compensatedprediction parts can generate one vector from a plurality of encodedvectors, and add the vector to the candidate vectors, as shown in FIG.18. While such an increase in the number of candidate vectors increasesthe amount of information to be processed, the precision of the directvector can be improved and hence the encoding efficiency can beimproved.

In this Embodiment 3, the candidates for the direct vector can bedetermined on a per slice basis. Information showing which vectorsshould be selected as candidates is multiplexed into each slice header.For example, there can be considered a method of, because the effect ofa temporal vector is low in a video which is acquired by panning acamera, removing temporal vectors from the selection candidates for sucha video, and, because the effect of a spatial vector is large in a videowhich is acquired by a fixed camera, adding spatial vectors to theselection candidates for such a video.

While the larger the number of candidate vectors is, the nearer to theoriginal image a prediction image can be generated, a balance betweenthe amount of information to be processed and the encoding efficiencycan be achieved by determining the candidates in consideration of thelocality of the video, such as excluding ineffective vectors from thecandidates in advance, in order to prevent the amount of information tobe processed from greatly increasing due to the increase in the numberof candidate vectors. Switching a vector between a candidate and anon-candidate is achieved by using, for example, a method of providingan ON/OFF flag for each vector, and defining only a vector whose flag isset to ON as a candidate. A motion vector which can be a selectioncandidate can be switched between a candidate and a non-candidate byusing each slice header or each header in an upper layer, such as eachsequence header or each picture header. Further, one or more sets ofmotion vectors each of which can be a selection candidate can beprepared, and an index showing each of the candidate sets can beencoded.

Further, a vector can be switched between a candidate and anon-candidate for each macroblock or each block to be encoded. Switchinga vector between a candidate and a non-candidate for each macroblock oreach block to be encoded can provide the macroblock or block to beencoded with locality, and provides an advantage of improving theencoding efficiency. Further, the selection candidates can be determineduniquely for each partition block size. Because the spatial correlationgenerally becomes weak as the block size becomes small, it can beexpected that the predictive accuracy of a vector determined through amedian prediction gets worse. To solve this problem, by removing amotion vector determined through a median prediction from thecandidates, for example, the amount of information to be processed canbe reduced without lowering the encoding efficiency.

Although the explanation is made in this Embodiment 3 by assuming thecase in which both a temporal direct vector and a spatial direct vectorexist, there is a case in which no motion vector exists when an intraencoding process is carried out on the block to be encoded B^(n). Inthis case, there can be considered a method of setting a zero vector asa motion vector, a method of not including any motion vector in thecandidates, and so on. While the encoding efficiency can be improvedbecause the candidates increase in number when a zero vector is set as amotion vector, the amount of information to be processed increases. Whenno motion vector is included in the candidates for direct vector, theamount of information to be processed can be reduced.

Although the example of generating a direct vector is shown in thisEmbodiment 3, the direct vector can be used as a predicted vector whichis used for encoding of a normal motion vector. While the amount ofinformation to be processed increases when the direct vector is used asa predicted vector, the encoding efficiency can be improved because theaccuracy of the prediction increases.

Although the example of calculating an evaluated value SAD from acombination of an image located backward of the block to be encodedB^(n) in time and an image located forward of the block to be encodedB^(n) in time is shown in this Embodiment 3 (refer to FIG. 15), anevaluated value SAD can be alternatively calculated from a combinationof only images located backward of the block to be encoded B^(n) intime, as shown in FIG. 19. As an alternative, an evaluated value SAD canbe calculated from a combination of only images located forward of theblock to be encoded B^(n) in time. In this case, temporal vectors areexpressed by the following equations (12) and (13).

$\begin{matrix}{{\hat{v}}_{0} = {\frac{d_{0}}{d_{col}}v_{col}}} & (12) \\{{\hat{v}}_{1} = {\frac{d_{1}}{d_{col}}v_{col}}} & (13)\end{matrix}$

where {circumflex over (v)}₀ is the vector of list 0, and {circumflexover (v)}₁ is the vector of list 1. In the above equations, d denotes atemporal distance, d₀ denotes the temporal distance of a list 0reference image, and d1 denotes the temporal distance of a list 0reference image. Further, v_(col) and d_(col) denote the vector of ablock spatially located at the same position in the reference image asthe block to be encoded, and the temporal distance of the referenceimage shown by the vector, respectively.

Even in a case in which the two reference image lists indicate the samereference image, the same method as that shown in FIG. 19 can beapplied.

Although the case in which each of the two reference image lists has twoor more reference images is assumed in this Embodiment 3, there can beconsidered a case in which only one reference image is included in eachof the two reference image lists. In this case, when the same referenceimage is set to the two reference image lists, there can be a case inwhich the determination can be carried out by using only a spatialvector without using any temporal vector. When different referenceimages are set to the two reference image lists, respectively, thedetermination can be handled by using the above-mentioned method.

Although a prediction process from two directions is assumed to becarried out in this Embodiment 3, a prediction process only in a singledirection can be alternatively carried out. When a prediction from avector in one direction is carried out, information showing which vectoris used is encoded and transmitted. As a result, a problem, such asocclusion, can be dealt with, and a contribution to an improvement inthe predictive accuracy can be made.

Although it is assumed in this Embodiment 3 that a prediction using twovectors is carried out, the number of vectors can be three or more. Inthis case, for example, there can be considered a method of generating aprediction image by using all vector candidates each of which providesan evaluated value SAD equal to or smaller than a threshold Th, among aplurality of vector candidates. Further, instead of using all candidateseach of which provides an evaluated value SAD equal to or smaller thanthe threshold Th, a maximum of the number of vectors which are used canbe preset to each slice header or the like, and a prediction image canbe generated by using the maximum number of vectors each of whichprovides a smaller evaluated value.

A vector is determined from an evaluation between reference images inthis Embodiment 3. This evaluation can be carried out from a comparisonbetween an already-encoded image which is spatially adjacent to theblock to be encoded and a reference image. In this case, there can beconsidered a method of carrying out the evaluation by using such anL-shaped image as shown in FIG. 20. Further, when an already-encodedimage which is spatially adjacent to the block to be encoded is used,there is a possibility that the already-encoded image is not in time forthe comparison because of pipeline processing. In this case, there canbe considered a method of using the prediction image instead of thealready-encoded image.

Although the example of searching for a motion vector after determiningan initial vector is shown in this Embodiment 3, whether or not tosearch for a motion vector by using a flag can be determined on a perslice basis. In this case, while the encoding efficiency is reduced,there is provided an advantage of being able to greatly reduce theamount of information to be processed. The flag can be provided on a perslice basis or can be determined for each sequence, each picture or thelike in an upper layer. When the flag is in an OFF state and no motionsearch is carried out, the same operation as that according toabove-mentioned Embodiment 1 is performed.

Although it is assumed in this Embodiment 3 that each of the directvector generating parts 25 and 64 carries out the vector generatingprocess regardless of the block size, this process can be limited to acase in which the block size is equal to or smaller than a predeterminedblock size. A flag showing whether or not to limit the process to thecase in which the block size is equal to or smaller than thepredetermined block size, and information showing the predeterminedblock size can be multiplexed into each header in an upper layer such aseach slice header. The flag and the information can be changed accordingto a maximum CU size. There is a tendency for the correlation betweenreference images to become low and for errors to become large as theblock size becomes small. Therefore, there are many cases in whichwhichever vector is selected, the performance is hardly affected, andthere is provided an advantage of reducing the amount of information tobe processed without reducing the encoding performance by turning offprocesses using large block sizes.

Embodiment 4

In above-mentioned Embodiment 1, the example in which each of themotion-compensated prediction parts 5 and 54 generates a spatial directvector in the spatial direct mode from the motion vector of analready-encoded block (already-decoded block) located in the vicinity ofthe block to be encoded and also generates a temporal direct vector inthe temporal direct mode from the motion vector of an already-encodedpicture (already-decoded block) which can be referred to by the block tobe encoded, and selects one direct vector which provides a highercorrelation between reference images from the spatial direct vector andthe temporal direct vector is shown. The motion-compensated predictionpart 5 of the moving image encoding device can alternatively select amotion vector suitable for the generation of a prediction image andcarry out a motion-compensated prediction process on the block to beencoded to generate a prediction image by using the motion vector, andcan also output index information showing the motion vector to thevariable length encoding part 13. On the other hand, themotion-compensated prediction part 54 of the moving image decodingdevice can alternatively carry out a motion-compensated predictionprocess on the coding block to generate a prediction image by using themotion vector shown by the index information which is multiplexed intothe bitstream.

FIG. 31 is a block diagram showing a motion-compensated prediction part5 of a moving image encoding device in accordance with Embodiment 4 ofthe present invention. In the figure, because the same referencenumerals as those shown in FIG. 2 denote the same components or likecomponents, the explanation of the components will be omitted hereafter.A direct vector generating part 26 carries out a process of referring toa direct vector candidate index in which a selectable motion vector andindex information indicating the motion vector are described to select amotion vector suitable for the generation of a prediction image from oneor more selectable motion vectors, and outputting the motion vectorselected thereby to a motion compensation processing part 24 as a directvector and also outputting the index information showing the motionvector to a variable length encoding part 13. Whenvariable-length-encoding compressed data, an encoding mode, etc., thevariable length encoding part 13 includes the index information in interprediction parameters and then variable-length-encodes these interprediction parameters.

FIG. 32 is a block diagram showing a motion-compensated prediction part54 of a moving image decoding device in accordance with Embodiment 4 ofthe present invention. In the figure, because the same referencenumerals as those shown in FIG. 6 denote the same components or likecomponents, the explanation of the components will be omitted hereafter.A direct vector generating part 65 carries out a process of receiving adirect vector candidate index in which a selectable motion vector andindex information showing the selectable motion vector are described,reading the motion vector shown by the index information included in theinter prediction parameters from the direct vector candidate index, andoutputting the motion vector to a motion compensation processing part 63as a direct vector.

Next, the operation of the moving image encoding device and theoperation of the moving image decoding device will be explained. Becausethe moving image encoding device and the moving image encoding deviceaccording to this embodiment have the same structures as those accordingto above-mentioned Embodiment 1, with the exception that the directvector generating parts 23 and 62 of the motion-compensated predictionparts 5 and 54 according to above-mentioned Embodiment 1 are replaced bythe direct vector generating parts 26 and 65, as compared withabove-mentioned

Embodiment 1, only processing carried out by each of the direct vectorgenerating parts 26 and 65 will be explained hereafter.

The direct vector generating part 26 of the motion-compensatedprediction part 5 generates a direct vector for each partition P_(i)^(n) of a block to be encoded B^(n) when the encoding mode m(B^(n)) ofthe block is a direct mode. More specifically, the direct vectorgenerating part 26 selects a motion vector suitable for the generationof a prediction image from one or more selectable motion vectors byreferring to the direct vector candidate index as shown in FIG. 33.Although five motion vectors are listed as the one or more selectablemotion vectors in the example shown in FIG. 33, an index of 0 isassigned to “median” in a space prediction because “median” is selectedmost frequently in the space prediction.

When selecting a motion vector suitable for the generation of aprediction image, the direct vector generating part 26 calculates a costR from the prediction image, which is acquired from each of theselectable motion vectors, the distortion of the original image, and theindex code amount of each of the selectable motion vectors, as shown inthe following equation (14), and selects the motion vector whose cost Ris the smallest from among the plurality of motion vectors.

R=min{D+λ _(i)ζ(i)}_(i=1, . . . n)   (14)

-   -   where D is the residual signal between the prediction image and        the original image, i is the index, λ is a Lagrange multiplier,        and λ( ) is the code amount of the term within the parentheses.

After selecting the motion vector whose cost R is the smallest fromamong the plurality of motion vectors, the direct vector generating part26 outputs the motion vector to the motion compensation processing part24 as a direct vector, and also outputs the index information indicatingthe motion vector to the variable length encoding part 13. For example,when selecting “median” as the motion vector whose cost R is thesmallest, the direct vector generating part outputs the index of 0 tothe variable length encoding part 13, whereas when selecting “MV_A” asthe motion vector whose cost R is the smallest, the direct vectorgenerating part outputs an index of 1 to the variable length encodingpart 13. When receiving the index information from the direct vectorgenerating part 26, the variable length encoding part 13 includes theindex information in the inter prediction parameters and thenvariable-length-encodes these inter prediction parameters whenvariable-length-encoding the compressed data, the encoding mode, etc.

When the encoding mode m(B^(n)) of the coding block B^(n) is a directmode, the direct vector generating part 65 of the motion-compensatedprediction part 54 generates a direct vector for each partition P_(i)^(n) of the coding block B^(n). More specifically, the direct vectorgenerating part 65 receives the same direct vector candidate index(e.g., the direct vector candidate index shown in FIG. 33) as that whichthe direct vector generating part 26 shown in FIG. 31 receives. Whenreceiving the inter prediction parameters including the indexinformation from a selection switch 61, the direct vector generatingpart 65 reads the motion vector shown by the index information from thedirect vector candidate index, and outputs this motion vector to themotion compensation processing part 63 as a direct vector. For example,when the index information is the index of 0, the direct vectorgenerating part outputs “median” as a direct vector, whereas when theindex information is the index of 1, the direct vector generating partoutputs “MV_A” as a direct vector.

As can be seen from the above description, because the moving imageencoding device in accordance with this Embodiment 4 is constructed insuch a way as to select a motion vector suitable for the generation of aprediction image from one or more selectable motion vectors and carryout a motion-compensated prediction process on a block to be encoded togenerate a prediction image by using the motion vector, and also outputindex information showing the motion vector to the variable lengthencoding part 13, there is provided an advantage of being able to selectan optimal direct mode for each predetermined block unit, thereby beingable to reduce the code amount, like in the case of above-mentionedEmbodiment 1.

Although the explanation is made in this Embodiment 4 by assuming thecase in which a motion vector exists at a selectable position, there isa case in which no motion vector exists when an intra coding process iscarried out on the block to be encoded B^(n). In this case, there can beconsidered a method of setting a zero vector as a motion vector, amethod of not including any motion vector in the candidates, and so on.While the encoding efficiency can be improved because the candidatesincrease in number when a zero vector is set as a motion vector, theamount of information to be processed increases. When no motion vectoris included in the candidates for direct vector, the amount ofinformation to be processed can be reduced.

Although the example of generating a direct vector is shown in thisEmbodiment 4, the vector can be used as a predicted vector which is usedfor encoding of a normal motion vector. While the amount of informationto be processed increases when the direct vector is used as a predictedvector, the encoding efficiency can be improved because the accuracy ofthe prediction increases.

Although the candidates for selectable motion vectors are fixed in thisEmbodiment 4, the candidates for selectable motion vectors can bealternatively determined on a per slice basis. Information showing whichvectors should be selected as the candidates is multiplexed into eachslice header. For example, there is a method of, because the effect of atemporal vector is low in a video which is acquired by panning a camera,removing temporal vectors from the selection candidates for such avideo, and, because the effect of a spatial vector is large in a videowhich is acquired by a fixed camera, adding spatial vectors to theselection candidates for such a video.

While the larger the number of candidate vectors is, the nearer to theoriginal image a prediction image can be generated, a balance betweenthe amount of information to be processed and the encoding efficiencycan be achieved by determining the candidates in consideration of thelocality of the video, such as excluding ineffective vectors from thecandidates in advance, in order to prevent the amount of information tobe processed from greatly increasing due to the increase in the numberof candidate vectors. Switching a vector between a candidate and anon-candidate is achieved by using, for example, a method of providingan ON/OFF flag for each vector, and defining only a vector whose flag isset to ON as a candidate. A motion vector which can be a selectioncandidate can be switched between a candidate and a non-candidate byusing each slice header or each header in an upper layer, such as eachsequence header or each picture header. Further, one or more sets ofmotion vectors each of which can be a selection candidate can beprepared, and an index showing each of the candidate sets can beencoded. Further, a vector can be switched between a candidate and anon-candidate for each macroblock or each block to be encoded. Switchinga vector between a candidate and a non-candidate for each macroblock oreach block to be encoded can provide the macroblock or block to beencoded with locality, and provides an advantage of improving theencoding efficiency.

Although the order of the indexes is fixed in this Embodiment 4, theorder of the indexes can be alternatively changed on a per-slice basis.When the selection of a vector which is carried out on a per-basis slicehas a bias, an index table is changed in such a way that a shorter codeis assigned to a vector having a higher selection frequency, therebyproviding an improvement in the encoding efficiency. Encoding ofinformation showing the change can be carried out by encoding the orderof each vector or by preparing a plurality of index sets and encodinginformation showing which index set is used. Further, there can beconsidered a method of predetermining only a default setting, preparinga flag showing whether or not to use a setting different from thedefault setting, and updating the index set and switching to the settingonly when the flag is set.

Although the example of changing the order of the indexes on a per slicebasis is shown above, it needless to say that the order of the indexescan be alternatively determined for each sequence, each picture or thelike in an upper layer. As an alternative, the order of the indexes canbe changed on a per partition block basis or on a per block to beencoded basis. Changing the order of the indexes on a per macroblockbasis or on a per block to be encoded basis can provide each macroblockor block to be encoded with locality, and can provide an improvement inthe encoding efficiency.

Further, the selection candidates can be determined uniquely for eachpartition block size. Because the spatial correlation generally becomesweak as the block size becomes small, it is considered that thepredictive accuracy of a vector determined through a median predictiongets worse. To solve this problem, by changing the order of the indexeswhich is determined through a median prediction, an improvement can beprovided in the encoding efficiency.

Although the direct vector candidate indexes respectively indicatingfive selectable motion vectors prepared in advance are shown in thisEmbodiment 4, six or more motion vectors or four or less motion vectorscan be prepared as the candidate vectors. For example, such vectorsclose to a temporal vector as shown in FIG. 17 and such a vectorresulting from a weighted sum of vectors in the vicinity of the block tobe encoded as shown in FIG. 18 can be added as candidate vectors.

Although a prediction process from two directions is assumed to becarried out in this Embodiment 4, a prediction process only in a singledirection can be alternatively carried out. When a prediction from avector in one direction is carried out, information showing which vectoris used is encoded and transmitted. As a result, a problem, such asocclusion, can be dealt with, and a contribution to an improvement inthe predictive accuracy can be made.

Although it is assumed in this Embodiment 4 that a bidirectionalprediction using two vectors is carried out, the number of vectors canbe three or more. In this case, for example, index information showingall the selected vectors can be encoded. In contrast with this, indexinformation showing vectors which are not selected can be encoded. As analternative, there can be considered a method of encoding only indexinformation showing a single vector, and using an image close to thereference image shown by the vector, as shown in FIG. 34.

Although the example of selecting a motion vector whose cost R is thesmallest from among a plurality of motion vectors is shown in thisEmbodiment 4, an evaluated value SADk can be calculated according to thefollowing equation (15) and a motion vector whose evaluated value SADkis equal to or smaller than a threshold Th can be selected.

SAD_(k) =|f _(index) −g _(k)|, 0≦k≦n   (15)

where f_(index) index denotes the reference image shown by the vectorwhose index information is encoded, and g_(k) denotes the referenceimage shown by a vector MV_k.

Although the example of using the evaluated value SAD_(k) is shownabove, it is needless to say that the evaluation is carried out by usinganother method such as SSE.

Information showing the number of vectors used can be multiplexed intoeach header in an upper layer, such as each slice header. While theencoding efficiency is improved with increase in the number of vectors,there is a trade-off relationship between the encoding efficiency andthe amount of information to be processed because the amount ofinformation to be processed increases with increase in the number ofvectors. As an alternative, the information showing the number ofvectors used can be multiplexed not into each slice, but into eachsmaller unit such as each block to be encoded or each partition. In thiscase, a balance can be achieved between the amount of information to beprocessed and the encoding efficiency according to the locality of theimage.

Although the example of selecting a motion vector suitable for thegeneration of a prediction image from among a plurality of selectablemotion vectors is shown in this Embodiment 4, a motion vector which isused as an initial vector can be selected from among a plurality ofselectable motion vectors, and, after that, a final motion vector can bedetermined by searching through the vicinity of the initial vector, likein the case of above-mentioned Embodiment 3. In this case, the directvector generating part 26 has a structure as shown in FIG. 35. Aninitial vector generating part 36 shown in FIG. 35 corresponds to theinitial vector generating part 34 shown in FIG. 26.

Embodiment 5

Each of motion-compensated prediction parts 5 and 54 in accordance withthis Embodiment 5 has the functions according to above-mentioned

Embodiment 1 (or Embodiment 2 or 3), and the functions according toabove-mentioned Embodiment 4, can switch between the functions accordingto above-mentioned Embodiment 1 (or Embodiment 2 or 3) and the functionsaccording to above-mentioned Embodiment 4 on a per slice basis, and canuse either of the functions according to above-mentioned Embodiment 1(or Embodiment 2 or 3) and the functions according to above-mentionedEmbodiment 4 to generate a prediction image.

FIG. 36 is a block diagram showing a motion-compensated prediction part5 of a moving image encoding device in accordance with Embodiment 5 ofthe present invention. In the figure, because the same referencenumerals as those shown in FIG. 31 denote the same components or likecomponents, the explanation of the components will be omitted hereafter.A direct vector generating part 27 carries out a process of, when adirect mode switching flag shows that index information is nottransmitted, generating a direct vector by using the same method as thatwhich the direct vector generating part 23 shown in FIG. 2 (or thedirect vector generating part 25 shown in FIG. 25) uses, and, when thedirect mode switching flag shows that index information is transmitted,generating a direct vector and also outputting index information showingthe direct vector to a variable length encoding part 13 by using thesame method as that which the direct vector generating part 26 shown inFIG. 31 uses. The direct vector generating part 27 also carries out aprocess of outputting the direct mode switching flag to the variablelength encoding part 13.

FIG. 37 is a block diagram showing the direct vector generating part 27which constructs the motion-compensated prediction part 5. Referring toFIG. 37, a selection switch 91 carries out a process of, when the directmode switching flag shows that the index information is not transmitted,outputting each partition P_(i) ^(n) of a block to be encoded B^(n) to apart corresponding to the direct vector generating part 23 shown in FIG.2 (or the direct vector generating part 25 shown in FIG. 25), and, whenthe direct mode switching flag shows that the index information istransmitted, outputting each partition P_(i) ^(n) of the block to beencoded B^(n) to a part corresponding to the direct vector generatingpart 26 shown in FIG. 31.

FIG. 38 is a block diagram showing a motion-compensated prediction part54 of a moving image decoding device in accordance with Embodiment 5 ofthe present invention. In the figure, because the same referencenumerals as those shown in FIG. 32 denote the same components or likecomponents, the explanation of the components will be omitted hereafter.A direct vector generating part 66 carries out a process of, when thedirect mode switching flag included in inter prediction parameters showsthat the index information is not transmitted, generating a directvector by using the same method as that which the direct vectorgenerating part 62 shown in FIG. 6 (or the direct vector generating part64 shown in FIG. 29) uses, and, when the direct mode switching flagshows that the index information is transmitted, generating a directvector by using the same method as that which the direct vectorgenerating part 65 shown in FIG. 32 uses.

Next, the operation of the moving image encoding device and theoperation of the moving image decoding device will be explained. Thedirect vector generating part 27 of the motion-compensated predictionpart 5 has the functions of the direct vector generating part 23 shownin FIG. 2 (or the direct vector generating part 25 shown in FIG. 25),and the functions of the direct vector generating part 26 shown in FIG.31, and, when the direct mode switching flag inputted thereto fromoutside the direct vector generating part shows that the indexinformation is not transmitted, generates a direct vector by using thesame method as that which the direct vector generating part 23 shown inFIG. 2 (or the direct vector generating part 25 shown in FIG. 25) uses,and outputs the direct vector to a motion compensation processing part24. The direct vector generating part 27 also outputs the direct modeswitching flag to the variable length encoding part 13.

When the direct mode switching flag shows that the index information istransmitted, the direct vector generating part 27 generates a directvector by using the same method as that which the direct vectorgenerating part 65 shown in FIG. 32 uses, and outputs the direct vectorto the motion compensation processing part 24. The direct vectorgenerating part 27 also outputs the direct mode switching flag and theindex information to the variable length encoding part 13.

When receiving the direct mode switching flag from the direct vectorgenerating part 27, the variable length encoding part 13 includes thedirect mode switching flag in the inter prediction parameters andvariable-length-encodes these inter prediction parameters whenvariable-length-encoding compressed data, an encoding mode, etc. Whenreceiving the direct mode switching flag and the index information fromthe direct vector generating part 27, the variable length encoding part13 includes the direct mode switching flag and the index information inthe inter prediction parameters and variable-length-encodes these interprediction parameters when variable-length-encoding the compressed data,the encoding mode, etc.

When receiving the inter prediction parameters decoded by a variablelength decoding part 51, the direct vector generating part 66 of themotion-compensated prediction part 54 generates a direct vector by usingthe same method as that which the direct vector generating part 62 shownin FIG. 6 (or the direct vector generating part 64 shown in FIG. 29)uses when the direct mode switching flag included in the interprediction parameters shows that the index information is nottransmitted. In contrast, when the direct mode switching flag shows thatthe index information is transmitted, the direct vector generating partgenerates a direct vector by using the same method as that which thedirect vector generating part 65 shown in FIG. 32 uses.

In general, additional information increases in a mode in which theindex information is transmitted as compared with a mode in which theindex information is not transmitted. Therefore, when the percentage ofthe additional information in the total code amount is large, such aswhen the transmission rate is low, the performance in a mode in whichthe index information is not transmitted is higher than that in a modein which the index information is transmitted. In contrast, when thepercentage of the additional information in the total code amount issmall, such as when the transmission rate is high, it is expected thatthe encoding efficiency is further improved by adding the indexinformation and using an optimal direct vector.

Although the example in which the direct mode switching flag is includedin the inter prediction parameters is shown in this Embodiment 5, thedirect mode switching flag can be multiplexed into each slice header,each picture, or each sequence header.

Further, there can be considered a method of determining the switchingaccording to the partition size. In general, the percentage of theadditional information, such as a motion vector, becomes smallrelatively with increase in the partition size. Therefore, there can beconsidered a structure of selecting a mode in which the indexinformation is transmitted when the partition size is equal to or largerthan a certain size, and, when the partition size is smaller than thecertain size, selecting a mode in which the index information is nottransmitted. When using the method of determining the switchingaccording to the partition size, as mentioned above, a flag showingwhich mode is used for each encoding block size can be multiplexed intoeach header in an upper layer, such as each slice header.

Although the example of switching between the functions according toabove-mentioned Embodiment 1 and the functions according toabove-mentioned Embodiment 4 according to the direct mode switching flagis shown in this Embodiment 4, switching between the functions accordingto above-mentioned Embodiment 2 and the functions according toabove-mentioned Embodiment 4 or switching between the functionsaccording to above-mentioned Embodiment 3 and the functions according toabove-mentioned Embodiment 4 can be alternatively carried out. As analternative, switching between the functions according toabove-mentioned Embodiment 1 and the functions according toabove-mentioned Embodiment 2, switching between the functions accordingto above-mentioned Embodiment 1 and the functions according toabove-mentioned Embodiment 3, or switching between the functionsaccording to above-mentioned Embodiment 2 and the functions according toabove-mentioned Embodiment 3 can be carried out. As an alternative,arbitrary functions can be selected from among the functions accordingto above-mentioned Embodiment 1 to 4.

Although the example of switching between the functions according toabove-mentioned Embodiment 1 and the functions according toabove-mentioned Embodiment 4 according to the direct mode switching flagis shown in this Embodiment 5, an ON/OFF flag can be provided instead ofswitching between the functions according to above-mentioned Embodiment1 and the functions according to above-mentioned Embodiment 4 accordingto the direct mode switching flag. For example, there can be considereda method of providing an ON/OFF flag showing whether or not to useEmbodiment 1, and, when the flag is set, carrying out both Embodiment 1and Embodiment 4 to select one mode which provides a higher degree ofencoding efficiency from the modes and encode the information. Thismethod provides an advantage of being able to switch between directmodes according to the locality of the image and make a contribution toan improvement in the encoding efficiency.

Although the flag for turning on or off Embodiment 1 is provided in theabove-mentioned example, a flag for turning on or off Embodiment 4 canbe alternatively provided. As an alternative, Embodiments 2 and 4 orEmbodiments 3 and 4 can be combined.

Although the example of selecting a motion vector suitable for thegeneration of a prediction image from a plurality of selectable motionvectors is shown in this Embodiment 5, a motion vector which is used asan initial vector can be selected from among a plurality of selectablemotion vectors, and, after that, a final motion vector can be determinedby searching through the vicinity of the initial vector, like in thecase of above-mentioned Embodiment 3. In this case, the direct vectorgenerating part 27 has a structure as shown in FIG. 39. An initialvector generating part 37 shown in FIG. 39 corresponds to the initialvector generating part 34 shown in FIG. 26.

While the invention has been described in its preferred embodiments, itis to be understood that an arbitrary combination of two or more of theabove-mentioned embodiments can be made, various changes can be made inan arbitrary component according to any one of the above-mentionedembodiments, and an arbitrary component according to any one of theabove-mentioned embodiments can be omitted within the scope of theinvention.

Although it is described above that, for example, a maximum size isdetermined and a hierarchy number upper limit on the number ofhierarchical layers in a hierarchy in which each of blocks to be encodedhaving the maximum size is hierarchically divided into blocks is alsodetermined, and an encoding mode which is suitable for each of theblocks to be encoded into which each block to be encoded having themaximum size is divided hierarchically is selected from one or moreavailable encoding modes, either or all of the maximum size, thehierarchy number upper limit, and the encoding mode can be alternativelydetermined in advance.

Embodiment 6

Although the example in which the direct vector generating part 26 ofthe motion-compensated prediction part 5 in the moving image encodingdevice grasps one or more selectable motion vectors by referring to adirect vector candidate index as shown in FIG. 33 is shown inabove-mentioned Embodiment 4, the encoding controlling part 1 canalternatively generate a list of one or more selectable motion vectorsaccording to the block size of a block to be encoded, and refer to thedirect vector candidate list showing the one or more selectable motionvectors and the direct vector candidate index to determine a direct modevector. Concretely, an encoding controlling part according to thisembodiment operates in the following way.

As mentioned above, while one or more selectable motion vectors can bedetermined uniquely for each of block sizes for partition, for example,there is a high correlation between the partition which is the block tobe encoded and an adjacent block when the partition has a large blocksize, whereas there is a low correlation between the partition which isthe block to be encoded and an adjacent block when the partition has asmall block size, as shown in FIG. 40. Therefore, the number ofcandidates for the one or more selectable motion vectors can be reducedwith decrease in the block size of the partition.

To this end, the encoding controlling part 1 lists one or moreselectable motion vectors in advance for each of the block sizesavailable for the partition which is the block to be encoded, as shownin FIG. 41. As can be seen from FIG. 41, the encoding controlling partreduces the number of candidates for the one or more selectable motionvectors with decrease in the block size of the partition. For example,while the number of selectable motion vectors is “4” for a partitionwhose block size is “64,” the number of selectable motion vectors is “2”for a partition whose block size is “8.” “median”, “MV_A”, “MV_B”,“MV_C”, and “temporal” shown in FIG. 42 correspond to “median”, “MV_A”,“MV_B”, “MV_C”, and “temporal” shown in FIG. 33, respectively.

When determining one or more selectable motion vectors, the encodingcontrolling part 1 refers to, for example, the list shown in FIG. 41,specifies the one or more motion vectors corresponding to the block sizeof the partition which is the target to be encoded, and outputs thedirect vector candidate list showing the one or more motion vectors to amotion-compensated prediction part 5. For example, when the block sizeof the partition is “64,” the encoding controlling part determines“MV_A”, “MV_B”, “MV_C”, and “temporal” as the one or more selectablemotion vectors. Further, when the block size of the partition is “8”,the encoding controlling part determines “median” and “temporal” as theone or more selectable motion vectors.

When receiving the direct vector candidate list from the encodingcontrolling part 1, a direct vector generating part 26 of themotion-compensated prediction part 5 selects a motion vector suitablefor the generation of a prediction image from the one or more motionvectors shown by the direct vector candidate list, like that accordingto above-mentioned Embodiment 4. In this case, because the number ofcandidates for one or more selectable motion vectors is small when theblock size of the partition is small, the number of calculations of anevaluated value SADk as shown in the above-mentioned equation (15), andso on is reduced and the processing load on the motion-compensatedprediction part 5 is reduced, for example.

In the case in which the encoding controlling part 1 of the moving imageencoding device determines one or more selectable motion vectors in thisway, a moving image decoding device also needs to have a list of one ormore selectable direct vector candidates which are the completely sameas those in the moving image encoding device. When the encoding modem(B^(n)) is a direct mode, for each partition P_(i) ^(n) of the codingblock B^(n), a variable length decoding part 51 of the moving imagedecoding device outputs the block size of the partition to amotion-compensated prediction part 54, and also outputs the indexinformation which the variable length decoding part acquires byvariable-length-decoding the bitstream (i.e., the information showingthe motion vector which is used by the motion-compensated predictionpart 5 of the moving image encoding device) to the motion-compensatedprediction part 54.

When receiving the block size of the partition from the variable lengthdecoding part 51, the direct vector generating part 65 of themotion-compensated prediction part 54 receives the direct vector indexand outputs the motion vector which is used for a direct mode from thelist of one or more motion vector candidates which is predeterminedaccording to the block size, like that according to above-mentionedEmbodiment 4. More specifically, the direct vector generating part 65lists one or more selectable motion vectors for each of the block sizesavailable for the partition in advance (refer to FIG. 41), and, whendetermining one or more selectable motion vectors, refers to the listshown in FIG. 41 and the direct vector index, and outputs the one ormore motion vectors corresponding to the block size of the partitionwhich is to be decoded this time.

For example, in a case in which the block size of the partition is “8”,the direct vector generating part outputs “median” as a direct vectorwhen the index information is an index of 0, and outputs “temporal” as adirect vector when the index information is an index of 1.

As can be seen from the above description, because the encodingcontrolling part in accordance with this Embodiment 6 is constructed insuch a way as to determine one or more selectable motion vectorsaccording to the block size of the partition which is the block to beencoded, a motion vector other than motion vectors suitable for thegeneration of a prediction image can be removed from the candidates fora partition having a low correlation between the partition and adjacentblocks. Therefore, there is provided an advantage of being able toreduce the amount of information to be processed.

Further, because the encoding controlling part in accordance with thisEmbodiment 6 is constructed in such a way as to, when determining one ormore selectable motion vectors, reduce the number of candidates for oneor more selectable motion vectors with decrease in the block size of thepartition, a motion vector other than motion vectors suitable for thegeneration of a prediction image can be removed from the candidates.Therefore, there is provided an advantage of being able to reduce theamount of information to be processed.

Although the example in which the block size of the partition which isthe block to be encoded has a maximum of “64” is shown in thisEmbodiment 6, the block size can alternatively have a maximum greaterthan 64 or less than 64. FIG. 42 shows an example of a list whosemaximum block size is “128.” Although the maximum block size of each ofthe lists held by the encoding controlling part 1 and themotion-compensated prediction part 54 is “128” in the example of FIG.42, a portion in which the block sizes are equal to or less than “32” inthe above-mentioned list has only to be referred to when the maximum ofthe block size of the actual partition is “32.”

Further, although the example of determining one or more selectablemotion vectors according to the block size of the partition which is theblock to be encoded is shown in this Embodiment 6, one or moreselectable motion vectors can be alternatively determined according tothe pattern of division of the block to be encoded, and the sameadvantages can be provided. FIG. 43 is an explanatory drawing of a listshowing one or more selectable motion vectors which are determined foreach of patterns of division available for the block to be encoded. Forexample, while “MV_A”, “MV_B”, “MV_C”, and “temporal” are determined asone or more selectable motion vectors when the partition which is theblock to be encoded is 2partH 1, there is a high possibility that whenthe partition which is the block to be encoded is 2partH2, its movementdiffers from that of 2partH1 which is the block located to the left of2partH2. Therefore, “MV_A” which is the motion vector of the blocklocated to the left of 2partH2 is removed from the one or more motionvectors selectable for 2partH2, and “MV_B”, “MV_C”, and “temporal” aredetermined as the one or more motion vectors selectable for 2partH2.

Further, although a vector in a temporal direction is used in thisEmbodiment 6, the data size of the vector when stored in a memory can becompressed in order to reduce the memory amount used for storing thevector. For example, when the minimum block size is 4×4, although avector in a temporal direction is typically stored for each block havinga size of 4×4, there is considered a method of storing a vector in atemporal direction for each block having a larger size.

A problem with the above-mentioned method of storing a vector in atemporal direction while compressing the data size of the vector is thatwhen carrying out the processing in units of a block having a block sizesmaller than the unit for storing the compressed vector data, theposition to be referred to does not indicate a correct position. Tosolve this problem, a process of not using any vector in a temporaldirection at a time when the block has a size smaller than the unit forstoring the compressed vector data can be carried out. By removing avector having a small degree of accuracy from the candidates, there isprovided an advantage of reducing the amount of information to beprocessed and the index code amount.

Further, although the direct mode vector is described in this Embodiment6, the same method can be used for the determination of a predictedvector which is uses for normal motion vector encoding. By using thismethod, there is provided an advantage of providing both a reduction inthe amount of information to be processed and an improvement in theencoding efficiency.

Further, this Embodiment 6 is constructed in such a way that whenref_Idx of a direct vector or a vector which is desired to be predicteddiffers from ref_Iclx of any of a plurality of candidate vectors whichare used for the generation of the direct vector or the determination ofthe predicted vector (the picture which is the reference destination ofthe direct vector or the vector to be predicted differs from that of anycandidate vector), a scaling process according to the distance in atemporal direction is carried out on each of the candidate vectors, asshown in FIG. 14. When ref_Idx of the direct vector or the vector whichis desired to be predicted is the same as ref_Idx of one of theplurality of candidate vectors, the scaling process according to thedistance in the temporal direction is not carried out.

$\begin{matrix}{{scaled\_ MV} = {M\; V\frac{d({Xr})}{d({Yr})}}} & (16)\end{matrix}$

where scaled_MV denotes a scaled vector, MV denotes a motion vector yetto be scaled, and d(x) denotes a temporal distance to x. Further, Xrdenotes the reference image shown by the block to be encoded, and Yrdenotes the reference image shown by each of the block positions A to Dwhich are the targets for scaling.

Further, this embodiment is constructed in such a way that a block whichis inter-encoded is searched for from the target blocks, and all thevectors included in the block are used as spatial vector candidates, asshown in FIG. 49.

There can be a case in which the reference picture which is to beindicated by the direct vector or the vector which is desired to bepredicted is the same as that indicated by one of these candidatevectors, and a case in which the reference picture which is to beindicated by the direct vector or the vector which is desired to bepredicted differs from that indicated by any of these candidate vectors,as mentioned above. In the former case, this embodiment can beconstructed in such a way that only candidate vectors indicating thesame reference picture are used as candidates. In the latter case, thisembodiment can be constructed in such a way that a correction process ofperforming a scaling process to make one of the candidate vectorsindicate the same reference picture is carried out. The former caseprovides an advantage of removing a vector having a low degree ofaccuracy from the candidates without increasing the amount ofinformation to be processed. The latter case provides an advantage ofreducing the code amount because the amount of information to beprocessed increases due to the search, but the number of selectioncandidates can be increased.

Further, in a case of carrying out scaling as shown in the equation(16), a candidate vector whose ref_Idx differs from ref_Idx of thedirect vector or the vector which is desired to be predicted can bescaled at a time of finding out a block which is inter-encoded (acandidate vector whose ref_Idx is the same as ref_Idx of the directvector or the vector which is desired to be predicted is not scaled), orthe scaling can be carried out only when there is no candidate vectorwhose ref_Idx is the same as ref_Idx of the direct vector or the vectorwhich is desired to be predicted after all the blocks are searchedthrough. Because a vector having an improved degree of accuracy can beadded to the candidates while the amount of information to be processedincreases, there is provided an advantage of reducing the code amount.

Embodiment 7

Although the example in which the encoding controlling part 1 of themoving image encoding device holds a list showing selectable motionvectors and the motion-compensated prediction part 54 of the movingimage decoding device also holds a list showing selectable motionvectors is shown in above-mentioned Embodiment 6, the variable lengthencoding part 13 of the moving image encoding device canvariable-length-encode list information showing the list and multiplexencoded data about the list information into, for example, each sliceheader, and transmit the encoded data to the moving image decodingdevice. In this case, the variable length decoding part 51 of the movingimage decoding device variable-length-decodes the encoded data which aremultiplexed into each slice header to acquire the list information, andoutputs the list shown by the list information to the direct vectorgenerating part 65 of the motion-compensated prediction part 54.

The moving image encoding device can transmit the list informationshowing the list to the moving image decoding device on a per slicebasis (or on a per sequence basis, on a per picture basis, or the like)in this way. As an alternative, only when the list currently being heldby the encoding controlling part 1 is changed, the moving image encodingdevice can transmit the list information showing the changed list to themoving image decoding device. Hereafter, processes will be explainedconcretely. FIG. 44 is a flow chart showing a transmitting process oftransmitting list information which is carried out by a moving imageencoding device according to this embodiment, and FIG. 45 is a flowchart showing a receiving process of receiving the list informationwhich is carried out by a moving image decoding device according to thisembodiment.

While an encoding controlling part 1 of the moving image encodingdevice, determines one or more selectable motion vectors according tothe block size of a partition which is a block to be encoded, like thataccording to above-mentioned Embodiment 6, the encoding controlling part1 checks to see whether the list to which the encoding controlling partrefers when determining one or more motion vectors is changed, and, whenthe list is the same as the previous list (step ST41 of FIG. 44), sets achange flag to “OFF” in order to notify the moving image decoding devicethat the list is the same as the previous list (step ST42). When theencoding controlling part 1 sets the change flag to “OFF”, a variablelength encoding part 13 encodes the change flag set to “OFF” andtransmits encoded data of the change flag to the moving image decodingdevice (step ST43).

In contrast, when the list differs from the previous list (step ST41),the encoding controlling part 1 sets the change flag to “ON” in order tonotify the moving image decoding device that the list differs from theprevious list (step ST44). When the encoding controlling part 1 sets thechange flag to “ON”, the variable length encoding part 13 encodes thechange flag set to “ON” and the list information showing the changedlist, and transmits encoded data of the change flag and the listinformation to the moving image decoding device (step ST45). FIG. 46shows an example in which the change flag set to “ON” and the listinformation showing the changed list are encoded because “temporal” inthe list is changed from selectable to unselectable.

A variable length decoding part 51 of the moving image decoding devicedecodes the encoded data to acquire the change flag (step ST51 of FIG.45), and, when the change flag is set to “OFF” (step ST52), outputs thechange flag set to “OFF” to a motion-compensated prediction part 54.When receiving the change flag set to “OFF” from the variable lengthdecoding part 51, the motion-compensated prediction part 54 recognizesthat the list is the same as the previous list and sets the listcurrently being held thereby as candidates for reference (step ST53).Therefore, the motion-compensated prediction part 54 determines one ormore motion vectors corresponding to the block size of the partitionwhich is to be decoded this time by referring to the list currentlybeing held thereby.

In contrast, when the change flag is set to “ON” (step ST52), thevariable length decoding part 51 of the moving image decoding devicedecodes the encoded data to acquire the list information and outputs thechange flag set to “ON” and the list information to themotion-compensated prediction part 54 (step ST54). When receiving thechange flag set to “ON” and the list information from the variablelength decoding part 51, the motion-compensated prediction part 54recognizes that the list differs from the previous list, changes thelist currently being held thereby according to the list information, andsets the list changed thereby as candidates for reference (step ST55).Therefore, the motion-compensated prediction part 54 determines one ormore motion vectors corresponding to the block size of the partitionwhich is to be decoded this time by referring to the list changedthereby. FIG. 47 shows an example in which the list currently being heldthereby is changed because the change flag is set to “ON.”

As can be seen from the above description, because the moving imageencoding device in accordance with this embodiment 7 is constructed insuch a way as to, only when a list showing one or more selectable motionvectors is changed, encode the list information showing the changed listto generate encoded data, there is provided an advantage of being ableto install a function of accepting a change of the list without causinga large increase in the code amount.

Although the example of, even when a part of the one or more selectablemotion vectors shown by the list is changed, encoding the listinformation showing the whole of the list changed is shown in thisEmbodiment 7, a change flag can be prepared for each block size, thechange flag prepared for a block size for which one or more selectablemotion vectors are changed can be set to “ON”, and only the listinformation associated with the block size can be encoded, as shown inFIG. 48. Because the motion vectors in a case of a block size of “64”and the motion vectors in a case of a block size of “8” are not changedin the example shown in FIG. 48, their change flags are set to “OFF” andthe list information associated with each of the block sizes is notencoded. In contrast, because the motion vectors in a case of a blocksize of “32” and the motion vectors in a case of a block size of “16”are changed in the example, their change flags are set to “ON” and thelist information associated with each of the block sizes is encoded. Aslong as the change flag of one of the block sizes is set to “ON,” thechange flag prepared for each block size can be encoded, and, when thechange flag of any block size is set to “OFF”, only the change flag ofthe list (change flag set to “OFF”) can be encoded. As an alternative,instead of using the change flag for each list, only the change flagprepared for each block size can be encoded.

Although the example of being able to change the selectable motionvectors for each block size is shown, the selectable motion vectors canbe changed for each pattern of division of the block to be encoded.

INDUSTRIAL APPLICABILITY

Because the moving image encoding device, the moving image decodingdevice, the moving image encoding method, and the moving image decodingmethod in accordance with the present invention make it possible toselect an optimal direct mode for each predetermined block unit andreduce the code amount, they are suitable for use as a moving imageencoding device, a moving image decoding device, a moving image encodingmethod, and a moving image decoding method which are used for an imagecompression encoding technology, an compressed image data transmissiontechnology, etc., respectively.

EXPLANATIONS OF REFERENCE NUMERALS

1 encoding controlling part (encoding controlling unit), 2 blockdividing part (block dividing unit), 3 selection switch (intraprediction unit and motion-compensated prediction unit), 4 intraprediction part (intra prediction unit), 5 motion-compensated predictionpart (motion-compensated prediction unit), 6 subtracting part(difference image generating unit), 7 transformation/quantization part(image compression unit), 8 inverse quantization/inverse transformationpart, 9 adding part, 10 memory for intra prediction, 11 loop filteringpart, 12 motion-compensated prediction frame memory, 13 variable lengthencoding part (variable length encoding unit), 21 selection switch, 22motion vector searching part, 23 direct vector generating part, 24motion compensation processing part, 25, 26, and 27 direct vectorgenerating part, 31 spatial direct vector generating part, 32 temporaldirect vector generating part, 33 direct vector determining part, 34,36, and 37 initial vector generating part, 35 motion vector searchingpart, 35 motion compensation part, 42 similarity calculating part, 43direct vector selecting part, 31 variable length decoding part (variablelength decoding unit), 52 selection switch (intra prediction unit andmotion-compensated prediction unit), 53 intra prediction part (intraprediction unit), 54 motion-compensated prediction part(motion-compensated prediction unit), 55 inverse quantization/inversetransformation part (difference image generating unit), 56 adding part(decoded image generating unit), 57 memory for intra prediction, 11 loopfiltering part, 12 motion-compensated prediction frame memory, 61selection switch, 62 direct vector generating part, 63 motioncompensation processing part, 64, 65, and 66 direct vector generatingpart, 71 spatial vector generating part, 72 temporal vector generatingpart, 73 initial vector determining part, 35 motion compensation part,82 similarity calculating part, 83 initial vector determining part, 91selection switch.

1. An image decoding device comprising: a variable length decoding unitfor performing variable-length decoding process on coded datamultiplexed into a bitstream to obtain compressed data, coding mode andindex information, each associated with a coding block; a motioncompensation prediction unit for performing a motion compensationprediction process on said coding block based on said coding mode togenerate a prediction image using a motion vector selected from one ormore selectable motion vector candidate, said motion compensationprediction unit selecting said motion vector indicated by said indexinformation; a decoded image generating unit for generating decodedimage data by adding a difference image which is obtained by decodingsaid compressed data and said prediction image generated by said motioncompensation prediction unit; and a loop filter for performing filteringprocess on said decoded image data; wherein said motion compensationprediction unit selects a spatial motion vector which is obtained from adecoded block located around said coding block or a temporal motionvector which is obtained from a decoded picture which can be referred toby said coding block according to said index information.
 2. An imagedecoding method comprising: a step for performing a variable-lengthdecoding process on coded data multiplexed into a bitstream to obtaincompressed data, coding mode and index information, each associated witha coding block; a step for performing a motion compensation predictionprocess on said coding block based on said coding mode to generate aprediction image using a motion vector selected from one or moreselectable motion vector candidate, said motion vector is selectedaccording to said index information; a step for generating decoded imagedata by adding a difference image which is obtained by decoding saidcompressed data and said prediction image; and a step for performingfiltering process on said decoded image data; wherein said motion vectorcandidate includes at least a spatial motion vector which is obtainedfrom a decoded block located around said coding block or a temporalmotion vector which is obtained from a decoded picture which can bereferred to by said coding block.